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VIRTUAL  ENVIRONMENTS  FOR  SOLDIER  TRAINING  VIA  EDITABLE 
DEMONSTRATIONS  (VESTED) 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Demonstrations  are  an  effective  and  highly  flexible  training  method  and  fulfill  an  essential  role 
in  military  training.  However,  although  they  are  recognized  as  an  effective  tool  to  train  key 
Army-relevant  capabilities,  there  is  little  detailed  guidance  on  how  to  best  generate  and  present 
effective  demonstrations.  Thus  it  is  clear  that  construction  of  effective  demonstrations  for 
training  Soldiers  would  benefit  from  more  complete  methodological  guidance  and  technological 
support  in  order  to  improve  the  effectiveness  and  efficiency  of  military  training.  VESTED 
(Virtual  Environments  for  Soldier  Training  via  Editable  Demonstrations)  is  a  Phase  II  SBIR 
effort  with  the  overarching  goal  of  aiding  the  production  of  effective  training  demonstrations. 


Procedure: 

Y ear  1  of  the  effort  focused  on  an  extensive  literature  review  examining  the  impact  of 
demonstration  design  features  on  training  effectiveness.  Year  1  work  also  included  the 
development  of  an  instructional  model  concept.  Additionally,  we  developed  and  articulated  a 
comprehensive  demonstration  authoring  use-model  and  system  architecture.  Finally,  we 
constructed  a  virtual  environment  concept  demonstration  as  an  application  case  to  inform  the 
development  of  the  system  and  authoring  guidelines. 

Y ear  2  of  the  effort  focused  on  how  to  aid  the  author  in  developing  detailed  connections  between 
demonstration  training  objectives  and  design  features.  A  series  of  semi-structured  interviews 
with  videographers,  documentary  filmmakers,  and  multimedia  specialists  was  conducted.  We 
selected  an  additional  application  case  to  further  focus  our  design  efforts.  Additionally,  we 
examined  the  literature  for  approaches  to  both  estimating  training  effectiveness  and  production 
costs.  Finally,  we  designed  and  partially  implemented  the  demonstration  authoring  tool. 


Findings: 

We  began  this  effort  with  the  expectation  we  would  be  developing  a  tool  that  would  aid  in  the 
development  of  demonstrations  through  a  process  based  on  empirical  guidelines.  Demonstration 
authoring  guidelines  were  developed,  however  with  the  understanding  that  some  of  the  relevant 
research  basis  for  this  guidance  may  be  ambiguous,  changing  over  time  and  conflicting.  Given 
these  guidelines  may  not  be  as  neat  and  generalizable  as  we  would  like;  we  concentrated  on 
identifying  the  task  requirements  of  the  training  demonstration  author.  VESTED  was  developed 
to  aid  in  appropriately  organizing  their  decisions  and  development  tasks,  all  based  on  the 
fundamental  assumption  that  understanding  and  specification  of  training  objectives  should  guide 


and  thread  through  all  aspects  of  demonstration  design.  VESTED  functionality,  as  driven  by  the 
need  to  articulate  design  objectives,  is  fundamental  to  the  achievement  of  effective 
demonstrations. 

Utilization  and  Dissemination  of  Findings: 

The  use  of  VESTED  should  yield  measurable  instructor,  training  content,  and  trainee  effects 
including  those  for  instructional  designers  (e.g.,  positive  instructor  reactions,  increased  instructor 
motivation),  with  regard  to  content  (e.g.,  increased  quality  of  demonstrations),  and  for  trainee 
performance  (e.g.,  increased  trainee  motivation).  The  use  of  VESTED  should  reduce  instructor 
workload  and  improve  instructor  efficiency  by  reducing  the  cost  of  developing  demonstrations 
and  permitting  demonstrations  to  be  executed  on  a  wide-variety  of  affordable  computer  hardware 
systems.  Moreover,  by  better  matching  instructional  content  to  trainee'  needs,  VESTED 
provides  a  medium  for  enhancing  trainee  motivation,  engagement  and  development  by  creating 
more  meaningful  learning  experiences. 
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VIRTUAL  ENVIRONMENTS  FOR  SOLDIER  TRAINING  VIA  EDITABLE 
DEMONSTRATIONS  (VESTED) 


This  technical  report  describes  the  design  and  development  activities  completed  through 
the  end  of  Year  2  for  the  VESTED  Phase  II  SBIR  effort.  The  purpose  of  VESTED  is  to  aid 
authors  of  demonstrations  in:  articulating  the  objectives,  requirements,  and  constraints  for  a 
planned  training  demonstration;  developing  one  or  several  storyboard  ‘cases’  that  would  satisfy 
those  conditions;  and  finally  structuring  the  cost-benefit  decision  of  selecting  a  preferred  case 
and  elaborating  its  specification  into  a  detailed  production  plan. 

As  the  U.S.  military  continues  to  redefine  its  role  to  include  participation  in  low  intensity 
conflicts,  asymmetric  warfare,  and  overseas  contingency  operations,  it  is  necessary  to 
continuously  redefine  tactical  doctrine,  and  the  training  of  that  doctrine,  based  on  lessons 
learned  in  the  field.  The  provision  of  effective  methods  and  technologies  for  training  of 
military  personnel  has  long  been  a  major  challenge  to  U.S.  military  forces,  and  recent  activities 
in  the  Middle  East  have  only  escalated  the  stakes  for  broader  ranges  of  training  content,  more 
reliable  training  results,  and  shorter  training  cycles  for  new  recruits  prior  to  overseas 
deployment. 

The  methods  and  processes  known  as  Instructional  Systems  Design  (ISD)  have  been 
developed  over  several  decades  to  address  military  needs.  In  particular,  the  1 1 -volume  MIL- 
HDBK-29612  addresses  a  broad  range  of  training  system  design  and  development  issues, 
including  a  nearly  300-page  Volume  2A  on  ISD.  For  the  broad  category  of  Knowledge,  Skills, 
and  Attitude  (KSA)  competencies  that  must  be  trained,  MIL-HDBK  29612  identifies  the 
following  sequence  of  instructional  components  as  appropriate:  (1)  Introduction,  (2)  Essential 
or  Core  Information,  (3)  Examples  or  Demonstration,  Practice  and  Feedback. 

Demonstrations  are  an  effective  and  highly  flexible  training  method  within  the  ISD 
framework.  Unfortunately,  however,  although  they  are  recognized  as  an  effective  tool  to  train 
key  Anny-relevant  capabilities,  there  is  little  detailed  guidance  on  how  to  best  generate  and 
present  effective  demonstrations.  The  complete  guidance  of  MIL-HDBK  29612  (p.  140)  is  as 
follows: 

Present  demonstrations.  Since  procedures  are  always  performed  the  same  way  on  the 
job,  only  one  example  of  how  the  procedure  is  applied  is  required.  A  demonstration  is  used  as 
a  concrete  example  of  how  a  procedure  should  be  performed  and  includes  explanations  of 
difficult  steps.  Procedures  may  be  performed  live  by  the  instructor,  on-line,  downloadable, 
presented  in  audiovisual  form,  or  appear  in  a  workbook.  Guidelines  for  developing  adequate 
demonstrations  are  as  follows: 

a.  Begin  with  a  description  of  the  specific  situation  in  that  the  procedure  will  be 
demonstrated.  Include  all  necessary  tools  and  equipment. 

b.  Cover  all  steps  in  the  order  presented.  Point  out  and  explain  common  errors. 
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c.  Indicate  all  steps  requiring  decisions  and  show  the  response  for  each  decision. 
Although  most  procedures  involve  a  set  of  steps,  all  of  which  are  performed  the  same  way 
every  time,  some  procedures  may  require  decision  steps  within  the  procedure.  Draw  the 
student’s  attention  to  these  steps.  This  can  be  done  by  using  the  "if...,  then  ..."  format. 

d.  Exclude  all  nonessential  information  from  the  demonstration. 

Thus,  it  is  clear  that  construction  of  effective  demonstrations  for  training  Soldiers  would 
benefit  from  more  complete  methodological  guidance  and  technological  support  in  order  to 
improve  the  effectiveness  and  efficiency  of  military  training. 

The  central  objective  of  this  Phase  II  SBIR  effort  is  to  produce  an  authoring  tool  and 
associated  guidance  to  create  effective  demonstrations  for  Army  training.  We  created  a 
demonstration  authoring  tool,  called  the  Virtual  Environments  for  Soldier  Training  via  Editable 
Demonstrations  (VESTED),  which  guides  an  author  through  a  demonstration  creation  process 
to  select  the  specific  learning  goals  to  be  demonstrated  and  to  construct  storyboards  depicting 
the  underlying  behaviors,  cognitive  decisions  and  tasks  being  demonstrated.  VESTED  also  aids 
the  author  in  making  the  decisions  about  where  and  how  to  use  the  virtual  environment  (VE) 
medium  and  all  of  the  other  relevant  authoring  tools. 

There  are  several  compelling  reasons  for  using  VE  representation  instead  of  the 
alternative  of  using  video  recording  of  live  actors  and  real  physical  environments  to  create 
demonstrations  for  Army  training.  The  chief  reason  is  that  VEs  can  be  much  more  easily 
controlled  and  edited  than  real  environments  (REs).  This  feature  is  especially  important  for  the 
experimental  research  that  is  needed  in  order  to  develop  more  nuanced  design  guidance  for 
demonstrations.  Specifically,  additional  research  is  warranted  to  parse  and  model  the  effects  of 
systematic  manipulations  of  demonstration  features  (e.g.,  level  of  detail,  perspective, 
representation  of  errors,  lighting,  highlighting,  etc.)  on  instructor,  trainee,  and  training  content 
criteria.  These  kinds  of  systematically  controlled  variations  of  a  broad  range  of  demonstration 
characteristics  can  be  accomplished  more  easily  and  cost-effectively  in  a  well-designed  VE 
authoring  tool  than  in  a  real  environment  facility  or  studio.  In  fact,  the  effectiveness  of  this 
type  of  VE  approach  is  documented  (Lampton,  McDonald,  Rodriguez,  Morris,  &  Parsons, 
2001).  Another  key  advantage  of  the  VE  authoring  approach  for  demonstrations  is  that  it  offers 
the  possibility  of  tighter  integration  with  the  use  of  VE  approaches  for  facilitating  the  skill 
practice  phase  of  training  in  the  same  instructional  sequence,  enabling  a  seamless  transition 
from  VE  demonstrations  in  early  phases  of  training  to  interactive  VE  exercises  of  the  same 
skills  in  later  phases.  Other  significant  advantages  of  the  VE  approach  derive  from  the 
considerable  investments  and  advances  in  this  area  that  both  Department  of  Defense  (DoD)  and 
the  commercial  serious  games  area  have  made  in  the  past  decade,  with  powerful  authoring  tools 
and  rich  libraries  of  object  imagery  currently  available  at  affordable  and  progressively 
decreasing  costs. 

VESTED  has  been  developed  specifically  to  facilitate  the  creation  of  demonstrations  for 
training  of  diverse  military  skills,  with  no  particular  attention  to  whether  or  not  the  skills 
required  interpersonal  interaction.  Although  the  intended  focus  was  on  the  VE  medium,  it  was 
clear  from  the  outset  that  a  major  challenge  for  the  training  product  author  is  to  determine  when 
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and  how  to  use  any  and  all  of  the  available  media  in  order  to  best  achieve  training  objectives  in 
a  cost-effective  fashion.  Although  we  expect  that  VE  will  be  optimal  for  many  applications,  it 
is  also  clear  that  the  established  media  of  video,  still  photography,  and  manual  artwork  will  also 
be  most  appropriate  for  many  cases  depending  on  many  factors.  For  the  VE  medium,  in 
particular,  it  is  clear  that  there  are  many  existing  and  diverse  software  tools  to  aid  the  author  in 
creation  of  products  such  as  training  videos. 

The  goal  of  VESTED  is  to  provide  the  author  with  an  organizing  tool  to  aid  them  in 
making  the  decisions  about  where  and  how  to  use  the  VE  medium  and  all  of  the  relevant  other 
authoring  tools.  The  basic  concept  of  VESTED  is  to  offer  a  tool  to  aid  the  author  in 
articulating  the  objectives,  requirements,  and  constraints  for  a  planned  training  demonstration, 
then  to  guide  them  through  the  development  of  one  or  several  storyboard  ‘cases’  that  would 
satisfy  those  conditions,  and  finally  to  structure  the  cost-benefit  decision  of  selecting  a 
preferred  case  and  elaborating  its  specification  into  a  detailed  production  plan. 

It  is  important  to  be  clear  about  the  definition  and  scope  of  the  term  demonstration  as  we 
use  it  in  reference  to  VESTED.  We  define  the  common  meaning  of  demonstration  as: 

An  observable  example  of  the  performance  of  a  task  by  one  or  more  persons  that  is 
presented  for  the  purpose  of  training  the  observer(s)  to  perform  the  task. 

It  will  be  useful  to  further  explore  the  meaning  of  some  of  the  elements  of  this  definition. 
We  most  often  think  of  demonstrations  as  visual  depictions  of  task  perfonnance,  though  sound 
and  especially  verbal  communications  involved  with  the  task  can  also  be  essential.  In  addition, 
it  is  critical  and  challenging  to  determine  how  to  handle  significant  but  inherently  unobservable 
aspects  of  task  performance,  particularly  cognitive  and  affective  aspects  that  are  not  normally 
verbalized.  These  issues  of  visibility  extend  to  the  consideration  of  possible  media  for 
presentation  of  the  demonstration,  which  include:  (1)  live  performance  by  actors/role-players 
that  are  physically  present  at  the  demonstration  site,  (2)  video  recording  of  live  actor 
performance,  (3)  video  screen  presentation  of  animated  actors  in  a  virtual  environment,  or  (4) 
some  hybrid  combination  of  these  elements. 

There  are  many  issues  to  consider  surrounding  the  exemplar  status  of  the  demonstration. 
Is  it  a  complete  activity  or  just  a  part?  Is  it  a  dynamic  representation  of  task  perfonnance  or 
just  a  static  snapshot?  Does  it  show  just  one  way  to  perform  the  task  correctly  or  multiple 
acceptable  alternatives?  Is  only  correct  perfonnance  exhibited,  or  are  some  typical  errors  in 
performance  also  included?  Particularly  significant  is  the  issue  of  whether  the  example  is  just 
the  raw  performance  of  the  task  or  whether  it  also  includes  additional  instructional  elements 
(e.g.,  tutorial  presentation)  that  are  provided  to  help  the  observer/trainee  understand  and  learn 
efficiently  from  the  demonstration.  For  example,  in  the  case  of  a  live  demonstration,  it  is 
typical  for  an  instructor  to  provide  an  instructional  envelope  around  a  demonstration,  including 
focused  comments  about  the  performance  example  before,  during,  and  immediately  after  the 
viewing  of  the  demonstration.  Training  videos  of  demonstrations,  both  from  live  actors  and 
animated  virtual  environments,  generally  provide  a  tutorial  envelope  around  the  raw 
performance  example  that  may  include  an  identification  of  the  training  objectives  for  the 
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demonstration,  comments  to  draw  attention  to  critical  or  difficult  aspects  of  performance,  and 
an  explanation  of  unobservable  cognitive  activities. 

Thus,  to  aid  the  production  of  effective  training  demonstrations  during  this  program,  we 
considered  the  following: 

a.  how  to  identify  the  appropriate  scope  for  the  role  of  the  demonstration  component 
within  a  complete  training  program  in  terms  of  what  specific  training  objectives  are  to  be  used 
to  establish  the  requirements  for  the  demonstration; 

b.  what  segments  of  task  performance  to  represent  in  the  demonstration; 

c.  whether  to  present  just  a  single  case  of  ideal  performance  or  possibly  multiple  cases 
that  may  represent  different  ways  of  doing  the  task; 

d.  whether  to  present  illustrations  of  common  or  dangerous  errors  in  task  perfonnance 
and  possibly  how  to  avoid  them; 

e.  whether  to  represent  that  performance  using  live  acting,  recordings  of  live  acting,  or 
animation  in  VEs; 

f.  how  to  select  and  use  available  relevant  technology  for  animation  and  VE  generation; 

g.  how  to  translate  training  objectives  and  other  design  decisions  into  script  and 
storyboard  for  production; 

h.  what  tutorial  material  and  other  production  tools  to  use  in  order  to  provide  a  complete 
instructional  envelope  for  the  demonstration;  and 

i.  how  to  factor  the  major  constraining  factors  of  cost  and  schedule  into  all  the  above 
decisions. 

It  is  important  that  the  VESTED  tool  is  highly  usable  by  the  intended  authors  so  that  they 
can  leam  to  use  it  quickly  and  they  can  generate  demonstrations  very  quickly  and  efficiently. 
The  design  of  the  tool  and  the  associated  methodology  integrated  into  the  tool  as  embedded 
guidance  and  help  must  ensure  that  all  generated  demonstrations  are  highly  effective  in 
achieving  their  training  goals.  By  better  matching  instructional  content  to  trainees’ 
developmental  needs,  VESTED  provides  a  medium  for  enhancing  trainee  motivation, 
engagement,  and  development  by  creating  more  meaningful  learning  experiences.  Thus, 
VESTED  should  have  proximal  effects  on  instructors,  which  translate  into  improved  training 
content  and,  ultimately,  more  effective  Soldiers. 
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VESTED  Phase  II  SBIR  Focus 


The  central  objective  of  this  Phase  II  SBIR  effort  is  to  produce  an  authoring  tool 
and  associated  guidance  to  create  effective  demonstrations  for  Army  training.  The 
overall  vision  for  the  VESTED  use  model  is  articulated  in  the  later  section  entitled 
“Vision  for  Overall  VESTED  Authoring  Use  Model.”  In  summary,  the  envisioned 
VESTED  functionality  can  be  broken  down  into  three  sequential  steps: 

1 .  Demonstration  specification  step  -  where  the  author  identifies  the  specific 
learning  objectives  to  be  modeled,  reviews  demonstration  authoring  guidelines,  creates  a 
set  of  demonstration  storyboards  that  reflect  the  target  behaviors,  and  then  chooses  a 
storyboard  to  fully  implement. 

2.  Demonstration  video  generation  step  -  whereby  the  demonstration  author 
utilizes  the  specific  game-based  environment  to  instantiate  the  behaviors  represented  in 
the  selected  storyboard  (thus  generating  a  set  of  raw  intermediate  videos). 

3.  Video  mixing  step  -  in  which  the  author  imports  intermediate  videos  into  a 
commercial-off-the-shelf  video  mixing  tool  and  performs  post-processing  to  create  the 
final  demonstration  video. 

We  began  this  effort  with  the  expectation  that  we  would  develop  a  software  tool 
that  would  support  all  the  functionality  outlined  above.  Our  initial  undertaking  was  to 
develop  the  demonstration  authoring  guidelines,  however,  with  the  understanding  that 
some  of  the  relevant  research  basis  for  this  guidance  may  be  absent,  ambiguous,  changing 
over  time,  and/or  conflicting.  Upon  concluding  that  these  guidelines  were  not  as  mature 
and  complete  as  we  desired  for  the  VESTED  concept,  we  determined  to  include  only  a 
few  general  guidelines  in  the  initial  version  of  VESTED  and  to  plan  for  expansion  and 
revision  of  those  guidelines  based  on  new  research  results  and  the  experiences  and 
evaluation  results  of  early  VESTED  applications.  For  VESTED  software  design,  we 
concentrated  on  identifying  the  task  requirements  of  the  training  demonstration  author 
and  developed  VESTED  to  aid  in  appropriately  organizing  their  decisions  and 
development  tasks,  all  based  on  the  fundamental  assumption  that  understanding  and 
specification  of  training  objectives  should  guide  and  thread  through  all  aspects  of 
demonstration  design.  VESTED  functionality,  as  driven  by  the  need  to  articulate  design 
objectives,  is  fundamental  to  the  achievement  of  effective  demonstrations.  Steps  2  and  3 
above  represent  areas  where  new  custom  VESTED  software  would  add  value  to  the 
overall  VESTED  tool;  however,  it  was  found  feasible  to  accomplish  those  steps 
according  to  VESTED  procedures  using  commercially  available  software  tools. 

Consequently,  during  this  effort,  VESTED  was  developed  to  offer  a  tool  that  aids 
the  author  in  articulating  the  learning  objectives,  requirements,  and  constraints  for  a 
planned  training  demonstration,  then  to  guide  them  through  the  development  of  one  or 
several  storyboard  ‘cases’  that  would  satisfy  those  conditions,  and  finally  to  structure  the 
cost-benefit  decision  of  selecting  a  preferred  case  and  elaborating  its  specification  into  a 
detailed  production  plan.  Thus,  the  current  VESTED  tool  concept  and  software  products 
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address  Step  1  of  the  above  VESTED  vision  and  the  output  of  that  step  includes  some 
detailed  guidance  for  Steps  2  and  3  for  which  future  extensions  of  VESTED  are  intended. 

Process  for  Guideline  Development 

This  section  details  our  approach  to  develop  demonstration  authoring  guidelines. 
The  first  step  is  to  describe  a  review  of  the  impact  of  demonstration  design  features  on 
training  effectiveness  as  referenced  in  the  literature.  We  then  present  our  work  on 
developing  a  VESTED  instructional  model  concept.  Next,  is  a  discussion  of  the 
formation  of  an  analytic  framework,  the  purpose  of  which  is  to  establish  a  reference  for 
the  specification  of  design  guidelines  for  training  demonstrations.  This  is  followed  by  a 
presentation  of  our  plan  and  findings  distilled  from  a  series  of  semi-structured  and 
unstructured  interviews  with  videographers,  documentary  filmmakers,  and  multimedia 
specialists  which  were  executed  to  better  understand  the  process  and  key  considerations 
inherent  in  developing  demonstrations.  Finally,  we  present  our  initial  collection  of 
instructional  design  guidelines  synthesized  from  our  review  of  the  literature  and  our  work 
on  the  analytical  framework. 

Literature  Review 

A  literature  search  was  conducted  using  online  and  library  resources,  starting  with 
publications  identified  in  the  Phase  I  research  and  the  Phase  II  proposal,  and  including 
the  network  of  citations  triggered  by  the  initial  tier  of  publications.  Electronic  searches 
of  computerized  databases  using  multiple  combinations  of  relevant  key  words  were 
executed.  More  specifically,  scholarly-oriented  search  engines  (e.g.,  Google  Scholar©), 
meta-databases  (e.g.,  EBSCOhost),  abstracting  services  (e.g.,  Dissertation  Abstracts 
International),  lessons  learned  repositories  (e.g.,  Defense  Technical  Information  Center) 
and  electronic  proceedings  (e.g.,  Proceedings  from  the  Interservice/Industry  Training 
Systems  and  Education  Conference  (I/ITSEC))  were  tapped  as  sources  of  valuable 
information  on  demonstrations.  The  second  step  of  the  literature  review  process  involved 
a  more  targeted  search  of  specific  academic  journals  known  for  consistently  publishing 
top-tier  training  research  (e.g.,  Journal  of  Applied  Psychology,  Personnel  Psychology, 
Military  Psychology,  Organization  Development  Journal,  Academy  of  Management 
Journal,  Small  Group  Research,  and  Human  Factors).  The  third  step  of  the  literature 
review  involved  ancestry  back-tracing  whereby  the  bibliographies  and  references  of 
already  identified  studies  were  searched  to  locate  earlier,  seminal  work  in  this  area.  The 
fourth  and  final  phase  of  the  four-step  literature  review  process  involved  reaching  out  to 
our  vast  network  of  colleagues  representing  multiple  scientific  disciplines  to  secure 
unpublished  articles,  technical  reports  and  conference  manuscripts. 

A  total  of  eighty-five  papers  of  significant  interest  were  identified  via  the 
literature  review;  abstracts  and  citations  were  obtained  for  all  of  them,  and  full  copies  for 
about  half.  Each  paper  was  then  classified  via  taxonomic  research  using  several 
taxonomies  (e.g.,  research  type)  and  associated  taxons  (e.g.,  survey  vs.  experimental) 
relevant  to  the  VESTED  project.  A  decision  rule  was  established  whereby  each  study 
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was  classified  into  a  single  taxon  by  its  overall  taxon  match.  The  following 
considerations  underpinned  the  taxonomic  effort: 

•  Application  area  requirements  and  product  evaluation 

•  Cost  analysis 

•  Dramatic  arts 

•  Developmental  processes  (i.e.,  maturation,  especially  in  adolescence) 

•  Guidelines  for  demonstrations 

•  Indus  try/ government  panel  reports 

•  Neurophysiology 

•  Parametric  experimental  studies 

•  Survey 

•  Scenario  Generation 

•  Sleep  effects  on  learning 

•  Sports  science 

•  Storyboarding 

•  Theory 

•  VE  technology 

Two  of  the  categories-parametric  studies  and  theory-were  found  to  be  quite 
broad  and  were  accordingly  each  broken  down  into  several  sub-categories.  For 
parametric  studies,  the  subcategories  included: 

•  Errors 

•  General  evaluation  of  demos 

•  Modality 

•  Observation  vs.  practice 

•  Perspective 

•  Sequence  and  repetition 

•  Timing 

For  publications  on  theory,  the  subcategories  included: 

•  AI  learning  theory  and  applications 

•  Behavior  Modeling  Training  (BMT) 

•  Cognitive 

•  Cognitive  load  theory 

•  General 

•  ISD  in  general 

•  Procedural  learning 
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Publications  are  listed  in  Appendix  A  and  organized  according  to  the  above- 
described  classification.  Not  surprisingly,  the  results  of  this  search  were  extensive  and 
diverse,  touching  on  all  of  these  disparate  issues  and  aspects  of  training  demonstrations. 
However,  we  were  able  to  identify  a  small  group  of  clusters  of  research  that  are 
especially  noteworthy.  These  are  briefly  discussed  in  the  following  paragraphs. 

The  U.S.  Army  Research  Institute  (ARI)  has  conducted  several  investigations  to 
address  the  training  effects  of  demonstrations.  A  study  by  Shlechter  and  Anthony  (1996) 
provided  an  extensive  exploration  of  the  general  problem  of  training  value  of 
demonstrations  and  evaluated  the  training  value  of  a  set  of  training  demonstration  videos 
produced  by  the  Anny  for  a  new  experimental  command  and  control  system.  The 
authors  point  out  that  explanations  in  demonstration  tapes  should  be  spoken  rather  than 
presented  as  written  text  because  of  cognitive  load  effects  on  visual  processing  of 
demonstration  visuals;  demonstrations  should  be  presented  several  times  to  achieve 
maximum  effect;  and  "cognitive  modeling"  (model  narration  of  thought  processes) 
should  be  provided  to  explain  cognitive  activity  accompanying  observable  actions.  The 
authors  found  the  specific  demonstration  videos  evaluated  to  be  valuable  for  enhancing 
training  in  combination  with  practice.  A  series  of  earlier  ARI  studies  by  Hagman  (1980a, 
b,  c)  are  summarized  below  under  the  category  of  motor  learning. 

Behavioral  Modeling  Training  (BMT).  The  area  of  work  referred  to  as  (BMT)  is 
derived  primarily  from  the  social  cognitive  theory  of  Bandura  (1977).  This  theory 
focuses  on  the  central  importance  of  observational  learning  processes  in  many  areas  of 
learning  and  behavior  modification,  both  intentional  and  unintentional.  Key  aspects  of 
the  observational  learning  processes  are  attention,  retention,  production,  and  motivation. 
The  majority  of  this  work  employs  live  actors  as  the  modelers  of  the  target  behaviors, 
with  much  of  the  research  and  applications  set  in  the  context  of  management  training. 
Much  of  this  work  is  relevant  to  key  issues  identified  for  VESTED.  Mayer  and  Russell 
(1987)  review  the  issues  of  variability  in  model  behavior  and  types  of  verbal  explanations 
(termed  “learning  points”  in  BMT).  Baldwin  (1992)  notes  specifically  that  there  may  be 
an  inverse  relationship  between  reproduction  and  generalization  in  BMT  with  non¬ 
variable  model  behaviors  facilitating  reproduction  of  those  precise  behaviors  by  the 
trainee  but  yielding  little  generalized  learning  while  more  variable  model  behaviors  may 
lead  to  slower  learning  of  the  ability  to  reproduce  the  target  behaviors  but  much  more 
effective  learning  of  generalization.  Taylor,  Russ-Eft,  and  Chan  (2005)  provide  a  meta- 
analytic  summary  of  work  in  the  overall  BMT  area. 

Experiment-based  cognitive  research.  Over  the  past  decade,  a  group  of 
researchers  have  conducted  systematic  theoretical  developments  and  experimental 
investigations  of  diverse  aspects  of  training  demonstrations  and  other  aspects  of  training 
pedagogy  as  implemented  using  the  various  media  and  tools  of  electronic  and  virtual 
applications,  such  as  simulation,  computer  games,  and  virtual  environments.  The  concept 
of  cognitive  load  limitations  as  key  factors  in  training  has  been  advanced  by  a  number  of 
researchers  (Chandler  &  Sweller,  1991;  Mayer  &  Moreno,  2003;  Paas,  Tuovinen, 

Tabbers,  &  van  Gerven,  2003).  Wouters,  Paas,  &  van  Merrienboer  (2008)  developed 
guidelines  for  observational  learning  applications  based  on  cognitive  load  theory,  with 


recommendations  in  the  three  areas  of  managing  subject  matter  complexity,  eliminating 
obstructions  to  learning,  and  stimulating  trainees  to  focus  on  active  processing  of  relevant 
information.  In  a  related  publication  (Wouters,  Paas,  &  van  Merrienboer,  2009),  they 
observe  that  spoken  explanations  are  superior  to  written  explanations  in  accompanying 
animations  only  when  there  is  no  encouragement  or  opportunity  to  focus  on  and  process 
the  explanations,  while  text  explanations  can  be  superior  when  trainees  are  given 
direction  and  opportunity  to  reflect  on  the  explanations. 

Sports  science  and  motor  behavior.  Possibly  the  largest  body  of  work  on  the 
subject  of  observational  learning  has  occurred  in  the  areas  of  motor  behavior,  especially 
with  applications  to  various  sports.  Ashford,  Bennet,  and  Davids  (2006)  offer  a  wide- 
ranging  meta-analytic  review  of  much  of  this  literature,  focusing  on  the  relative  effect 
sizes  for  criteria  in  the  areas  of  movement  dynamics  and  movement  outcomes.  A  series 
of  studies  by  Hagman  (1980a,  b,  c)  considered  the  training  effectiveness  of  demonstration 
presentations  of  a  simple  psychomotor  task,  examining  the  effects  of  variations  in  the 
sequence  of  demonstration  and  practice  events  relative  to  acquisition  speed  and 
subsequent  recall/decay.  Results  indicated  that  repeated  presentation  with  little  practice 
produced  much  poorer  retention  than  minimal  presentation  and  much  practice,  and  that 
alternating  presentation  and  practice  was  best  for  errors  in  acquisition  and  short-term 
retention  but  was  intennediate  between  mostly  practice  (best)  and  mostly  presentation 
(worst)  for  longer-term  retention.  Although  studies  in  this  area  have  addressed  diverse 
aspects  of  motor  behaviors  and  types  of  individual  and  group  sports,  one  fairly  consistent 
and  basic  message  seems  to  be  that  demonstration  combined  with  visual  and  verbal  cues 
produces  better  performance  than  practice  only  or  any  component  cue  alone  (e.g.,  Janelle, 
Champenoy,  Coombes,  &  Mousseau,  2003). 

Neurophysiology.  Recent  neurophysiological  research  has  revealed  the  existence 
of  brain  structures  that  seem  to  be  specifically  adapted  for  the  perfonnance  of 
observational  learning,  enabling  the  individual  to  understand  observation  of  the  behaviors 
of  other  individuals  in  terms  of  the  individual’s  own  behavioral  capabilities.  Petrosini, 
Graziano,  Mandolesi,  Neri,  Molinari  and  Leggio  (2003)  report  that  the  cerebellum  is 
critical  for  the  acquisition  of  complex  behaviors  by  observation,  but  is  not  so  much 
involved  in  fluent  performance  thereafter.  Frey  and  Gerry  (2006)  describe  a  mirror 
system  (inferior  frontal  and  parietal  cortices)  which  shows  increased  activity  during 
observation  of  a  model  and  greater  activity  when  the  observer  intends  to  replicate  the 
model's  action,  with  activity  in  the  intraparietal  sulcus  especially  critical.  Rizzolatti, 
Fogassi,  and  Gallese  (2006)  identify  specific  “mirror  neurons”  in  the  premotor  cortex  of 
monkeys  that  are  specifically  activated  during  imitative  behaviors  and  note  that 
functional  Magnetic  Resonance  Imaging  (fMRI)  results  indicate  similar  activation  in 
correlating  human  brain  regions  for  human  imitative  behaviors.  In  a  more  speculative 
vein,  Ramachandran  and  Oberman  (2006)  suggest  the  possibility  that  dysfunction  in  these 
mirror-neuron  structures  could  be  a  partial  cause  of  some  forms  of  autism  by  producing 
failures  in  the  capability  to  understand  the  behaviors  of  others  and  relate  them  to  one’s 
own  behaviors. 
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Dramatic  arts.  There  are  several  noteworthy  recent  publications  by  one  researcher 
that  concern  the  applications  of  principles  and  techniques  in  the  dramatic  arts  for 
enhancing  the  engagement  and  training  effectiveness  of  educational  games  and  other 
instructional  products  (El-Nasr,  2004,  2005,  2007).  Hopefully,  this  type  of  work  will  be 
expanded  in  the  near  future  to  provide  guidance  for  all  of  the  aspects  of  training 
demonstrations  that  are  understood  to  influence  the  success  of  the  training  experience  but 
do  not  seem  to  lend  themselves  to  systematic  experimental  study  (e.g.,  lighting, 
transitions,  perspective  shifting,  background  music,  etc.). 

This  brief  summary  of  several  key  areas  of  research  literature  pertaining  to  the 
design  of  training  demonstrations  has  led  us  to  the  conclusions  that  there  is  a  very  large 
body  of  relevant  work  reflecting  diverse  directions  of  inquiry,  and  much  significant  new 
work  should  be  expected  in  the  near  future,  especially  in  dynamic  new  areas  such  as  the 
connections  of  cognitive  experimentation  and  neurophysiology  via  techniques  like  fMRI. 
However,  it  has  also  become  apparent  that  many  detailed  issues  regarding  the 
effectiveness  of  specific  demonstration  design  features  seem  to  be  dependent  on  factors 
such  as  task  type,  learning  criteria,  and  other  features  of  context.  The  issue  of  whether 
and  how  to  represent  common  errors  has  been  of  particular  concern  in  this  regard. 
Shlechter  &  Anthony  (1996)  recommended  that  models  should  be  shown  making 
mistakes  but  also  indicated  that  the  research  literature  was  inconclusive  on  this  point.  For 
example,  on  a  medical  hand-washing  task,  Riolo  (1997)  found  no  differences  in  the 
training  effectiveness  of  demonstrations  with  or  without  errors.  Of  course,  failure  to  find 
an  expected  effect  could  be  the  result  of  either  an  insensitive  experimental  procedure  or 
the  absence  of  a  real  effect  (or  both).  Still,  in  many  applications  to  more  complex  skills, 
it  is  typical  to  provide  modeling  of  common  errors  and  even  to  evaluate  alternative 
techniques  to  insure  that  the  trainee  recognizes  the  displayed  erroneous  behaviors  (e.g., 
Jentsch,  Bowers  &  Salas,  2001).  The  lesson  for  a  design  tool  like  VESTED  seems  to  be 
to  provide  a  mechanism  for  offering  design  guidance  to  an  author  of  a  training 
demonstration  with  the  understanding  that  some  of  the  relevant  research  basis  for  that 
guidance  may  be  ambiguous,  changing  over  time,  and  even  with  individual  studies 
conflicting  with  one  another  in  some  cases,  thus  requiring  the  user  to  resolve  the 
appropriate  interpretations  of  the  emerging  research  picture  for  each  application  context 
while  awaiting  more  complete  consensus  guidance. 

Instructional  Framework  Concept 

Training  is  a  planned  activity  executed  to  specifically  target  a  trainee’s  or 
collective’s  latent  capacities  and  capabilities  as  well  as  observable  perfonnance  processes 
for  development.  In  order  for  investments  in  training  technologies  (VESTED)  to  translate 
in  to  organizationally  valued  outcomes  (increased  use  of  safe  practices),  a  chain  of  effects 
must  unfold  over  time;  preferably  in  accordance  with  a  well-conceived  theory.  To  more 
fully  illuminate  the  anticipated  effects  and  benefits  of  the  VESTED  solution,  we 
formalized  our  theory  of  VESTED’s  effects  by  leveraging  the  science  of  training  to 
specify:  (1)  what  effects  are  expected  from  the  system,  (2)  how  the  effects  are  related  in  a 
coherent  network  of  causal  relations  over  time,  and  (3)  the  conditions  under  which  the 
effects  are  and  are  not  interrelated  (see  Figure  1). 
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Figure  1 .  VESTED  effects  framework. 

As  is  illustrated  in  Figure  1,  it  is  anticipated  that  those  units  institutionalizing  VESTED 
will  set  in  motion  measurable  changes  in  instructors’  knowledge,  skill,  and  attitude  (KSA)  and 
thereby  trainees’  KSAs;  as  mediated  by  enhanced  demonstration  content.  Moreover,  over  time, 
the  introduction  of  VESTED  should  enhance  performance  transfer  from  the  training  context  to 
the  field  and  ultimately  to  positively  affect  the  larger  collectives  (crews,  squads,  platoons)  of 
which  trainees  are  members. 


By  articulating  a  substantive  theory  of  the  treatment  (VESTED),  and  detailing  its 
expected  relationships  with  mediating  processes  and  proximal  and  distal  outcomes,  we  have 
clearly  stated  the  anticipated  benefits  of  VESTED,  as  well  as  mapped  the  experiment-based 
multivariate  research  required  to  substantiate  these  assertions. 

In  addition  to  fostering  common  ground  on  VESTED’s  potential  benefits,  and  shedding 
light  on  how  to  determine  whether  the  articulated  benefits  accrue  via  systematic  research  efforts, 
Figure  1  also  provides  insight  about  how  VESTED  functions,  what  features  to  modify  should  it 
yield  less  than  optimal  results,  and  how  to  encode,  communicate  and  capitalize  on  the  formative 
and  summative  findings  yielded  via  training  effectiveness  evaluations  of  VESTED. 


The  Evolution  of  Expertise 

The  central  goal  of  training  is  generally  envisioned  as  transfonning  the  actual  trainee  into 
the  ideal  trainee  where  ideal  is  defined  by  targeted  competency  and/or  performance  levels  as 
established  by  instructors  or  other  training  subject  matter  experts  (SMEs).  Thus,  both  the  actual 
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and  ideal  trainees  can  be  represented  as  KSA  and  performance  models,  delineating  the  latent 
knowledge,  skill,  and  affective  competencies,  as  well  as  the  manifest  taskwork  and  teamwork 
performance  processes  that  enable,  drive,  and  culminate  in  perfonnance  outcomes  often 
ultimately  evaluated  in  terms  of  their  effectiveness  by  key  stakeholders.  Training  then  consists 
of  the  various  manipulations  that  are  contrived  to  effect  the  desired  transformation  in  latent 
capacities  and  capabilities  as  well  as  manifest  performance  processes,  typically  consisting  of 
some  combination  of  tutorial,  demonstration,  interactive/experiential  exercise,  evaluation,  and 
feedback  components.  Training  of  most  complex  skills  is  expected  to  employ  components  in  all 
of  these  areas,  sequenced  and  repeated  to  iteratively  achieve  desired  effects. 

An  idealized  method  for  optimal  training  is  to  develop  detailed  characterizations  of  the 
actual  student  model  and  desired  student  model,  to  then  derive  the  differences  between  the  two 
models  that  constitute  the  changes  that  must  be  effected,  and  then  to  select  and  apply  the  most 
appropriate  training  manipulations  to  produce  those  changes.  The  basic  process  representation 
envisioned  here  is  illustrated  in  Figure  2,  which  presents  the  concepts  of  the  Actual  Student 
Model  (ASM)  and  the  Ideal  Student  Model  (ISM)  as  the  foundational  elements  of  this  view  of 
training,  with  each  delineating  representations  of  each  of  the  various  aspects  of  declarative 
knowledge,  procedural  knowledge,  and  perceptual  and  motor  skills.  The  Difference  Student 
Model  (DSM)  represents  the  assessed  component  differences  between  the  ASM  and  ISM, 
differences  that  are  to  be  reduced  or  eliminated  through  the  training  process. 


Actual  Student  Model  (ASM) 

Ideal  Student  Model  (ISM) 

Declarative  Knowledge  (DK) 

Declarative  Knowledge  (DK) 

Terminology 

Terminology 

Concepts 

Concepts 

Principles 

Principles 

Procedural  Knowledge  (PK) 

Procedural  Knowledge  (PK) 

Perceptual  &  Motor  Skills  (P&MS) 

Perceptual  &  Motor  Skills  (P&MS) 

Difference  Student  Model  (DSM) 


ADK 


APK 


AP&MS 


Figure  2.  Concepts  of  actual,  ideal,  and  difference  student  models. 
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We  envision  the  KSA  map  of  the  trainee  to  represent  a  complex  network  of  declarative 
knowledge,  procedural  knowledge,  perceptual-motor  skills  and  problem-solving  skills  that  are 
relevant  to  the  perfonnance  of  the  tasks  being  addressed.  We  can  then  compare  the  KSA  map  of 
the  actual  student  with  that  of  the  ideal  student  in  each  of  these  component  areas  and  determine 
the  differences  that  need  to  be  addressed.  Additional  detail  about  the  KSA  constellation 
comprising  the  ISM  is  provided  next. 

Trainee-Centered  Training 

The  positive  and  negative  models  depicted  in  VESTED-based  demonstrations,  as  well  as 
the  narrative  descriptions  they  encompass,  are  advanced  as  levers  instructors  can  use  to  close 
gaps  between  the  ASM  and  ISM.  This  process  may  be  best  illustrated  by  an  example  from 
Military  Operations  on  Urban  Terrain  (MOUT);  the  first  use-case  of  VESTED  selected  by  the 
research  team.  When  considering  the  MOUT  domain,  the  visual  demonstration  of  the  correct 
procedure  (i.e.,  positive  model)  for  short-stocking  an  M-16  rifle  via  VESTED  should  enhance 
trainee  learning  of  MOUT  tactics,  techniques,  and  procedures  (TTPs).  Figure  3  depicts  the  role 
of  repeating,  varying,  and  cognitively  rehearsing  demonstrations  of  short-stocking  in  affecting  a 
trainee’s  motivation  to  learn  MOUT  TTPs.  We  suggest  that  highly  motivated  trainees  are  more 
likely  to  allocate  their  attention  and  self-regulatory  resources  to  learning  than  their  relatively  less 
motivated  teammates. 


Figure  3.  VESTED  MOUT  demonstration  student  effects  model. 
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The  establishment  of  goals,  and  the  expense  of  effort  in  goal  pursuit,  enables  the  process 
of  learning  by  which  trainees  acquire  MOUT  expertise.  Figure  3  depicts  our  conceptualization 
of  MOUT  expertise  in  terms  of  the  MOUT  ISM  (as  illustrated  in  the  dotted  line  box).  We 
suggest  trainees  subjected  to  demonstrations  who  are  motivated,  attentive,  and  actively  engaged 
in  the  learning  process  will  be  better  positioned  to  acquire  the  knowledge,  mental  models  and 
skills  required  to  effectively  enact  the  perfonnance  processes  (e.g.,  communication, 
coordination,  collaboration)  that  underpin  successful  urban  operations.  Over  time,  properly 
executed  performance  processes  will  culminate  in  training  outcomes  (targets  acquired,  obstacles 
safely  navigated,  etc.)  that  are  more  objective  indicators  of  learning.  The  capacities  and 
capabilities  reaped  from  exposure  to  VESTED  demonstrations  should  ultimately  transfer  to 
mission  operations,  enhancing  a  given  trainee’s  performance  and  also  that  of  his/her  team,  squad, 
platoon,  company,  and  battalion  as  illustrated  by  vertical  transfer  in  Figures  1  and  3. 


Development  of  an  Analytic  Framework  to  Structure  Design  Guidance 


This  section  addresses  the  requirement  to  create  an  analytic  framework  that  will  allow  us 
to  interconnect  specifications  of  training  objectives,  trainee  characteristics,  and  training  program 
characteristics  as  a  foundation  for  the  formulation  of  training  design  guidelines  for  training 
demonstrations  and,  more  generally,  for  complete  training  programs.  We  have  identified  the 
following  taxonomies  as  relevant  for  each  of  these  task,  trainee  and  training  program 
characteristic  component  areas: 

Task  Characteristics 
Modality 

Cognitive 

Motor 

Perceptual 

Visual 

Auditory 

Verbal 

Complexity 

Unordered  or  weakly  ordered  steps  or  actions 
Simple  linear  order 
Re  quiring  branching 

Amount  of  branching  required 
Conditions  for  branching 
Perception 
Communication 

Judgment  (decision,  complex  cognitive  determination) 

Agent  numbers 

Performer 

Single  agent  (solitary  individual  performer) 

Multiple  cooperating  agents  (teamwork  situation) 

Non-cooperating  contextual  agents 
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None 

Single  other  agent 
Multiple  other  agents 


Flexibility 

Rigid  (task  must  be  performed  precisely  as  specified) 

Variability  is  tolerated  but  not  necessary 

Variability  is  required  in  order  to  adapt  to  varying  conditions  of  context 
Scaffolding  on  other  KSA 

Prerequisite  KSA  (from  selection  or  prior  training) 

Integral  component  KSA 
Motor  skills 
Perceptual  skills 
Cognitive  skills 

Terminology 

Concepts 

Performance  criteria 
Retention 
Automaticity 
Transferability 
Timing 

Verbalizability 
Trainee  Characteristics 

Prior  relevant  knowledge  and  skills  in  all  relevant  areas  and  criteria 
None 

Some,  mainly  correct 

Some,  mainly  incorrect  and  counterproductive 
Relevant  cultural  background/context  (Hofstede  dimensions) 

Power  distance  index  (deference  to  organizational  hierarchy) 

Individualism  vs.  collectivism 
Masculinity  vs.  femininity 
Uncertainty  avoidance  index 
Long-term  orientation 

Specific  error  tendencies  that  should  be  countered  through  training 
None 

Some,  mostly  the  same  across  trainees 
Some,  highly  variable  across  trainees 
Individual  characteristics  vs.  population  distribution  statistics 

Since  we  must  assume  that  this  framework  will  require  us  to  separately  examine  each  cell 
in  the  intersection  of  all  of  the  orthogonal  categories,  the  total  number  of  cells  to  be  considered  is 
clearly  very  large.  However,  it  is  clear  that  not  all  of  these  cells  deserve  equal  attention.  In 
addition,  we  should  be  able  to  greatly  simplify  the  multiplicative  explosion  of  cells  in  the 
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framework  by  identifying  many  cases  where  there  are  good  reasons  to  assume  that  interactions 
between  categories  are  likely  to  be  negligible  or  otherwise  uninteresting.  Indeed,  it  seems 
unlikely  that  we  should  need  to  consider  interactions  between  most  of  the  trainee  characteristics 
categories  and  any  of  the  other  category  dimensions.  The  main  exceptions  in  this  area  would 
seem  to  be  considerations  of  how  tendencies  to  make  errors  should  be  considered  relative  to 
demonstrations  of  errors  and  possibly  how  some  other  demonstration  features  might  interact  with 
the  novice-expert  dimension  of  trainee  experience. 

The  purpose  of  the  proposed  framework  is  to  establish  a  reference  for  the  specification  of 
design  guidelines  for  training  demonstrations.  To  a  large  degree,  we  expect  that  the  formulation 
of  the  design  guidelines  will  occur  almost  as  a  collateral  aspect  of  the  identification  of  the 
framework  cells.  Essentially,  the  cells  are  defined  as  the  combinations  of  categories  for  training 
objective,  trainee  characteristics,  and  demonstration  design  features  for  which  a  single  design 
guideline  (or  coordinated  set  of  guidelines)  can  be  formulated.  By  revisiting  the  results  of  our 
earlier  literature  review,  we  identified  both  the  conditions  that  are  appropriate  for  the  definition 
of  framework  cells  and  the  guidelines  that  are  warranted  for  those  cells.  The  literature 
concerning  Army  applications  also  provided  indications  of  relative  priorities  for  the  need  for 
guidelines  in  many  of  the  areas  where  existing  research  literature  is  missing  or  ambivalent.  The 
scale  of  the  required  effort  in  this  regard  is  detennined  much  more  by  the  number  of  design 
features  of  interest  (on  the  order  of  a  dozen)  and  of  publications  that  must  be  re-examined  (on  the 
order  of  100),  with  only  a  few  publications  that  are  pertinent  to  each  design  feature,  rather  than 
the  number  of  potential  cells  in  the  framework  (in  the  tens  of  thousands),  so  it  should  be 
understood  to  be  a  practically  feasible  undertaking. 


Videographer  Interview  Process  &  Qualitative  Analysis 


A  series  of  structured  and  unstructured  interviews  were  conducted  with  subject  matter 
experts  (SME)  videographers  and  multimedia  specialists  including  a:  (1)  Director  and  Producer, 
(2)  Director,  (3)  Associate  Director,  (4)  Director  of  Post  Production,  and  (5)  Associate  Producer. 
All  but  one  of  the  five  interviewees  also  serves  as  an  Instructor  at  an  institution  of  advanced 
learning.  The  interviews  were  executed  to  better  understand  the  processes  and  key  considerations 
inherent  to  scoping,  storyboarding,  and  developing  documentary  films,  training  and 
informational  videos,  commercials,  and  multimedia  content.  The  purpose  of  this  process  was  to 
delineate  insight  about  functions  and  features  that  would  enhance  the  VESTED  software 
application.  In  addition  to  infonning  the  VESTED  design  process,  interviewee  feedback  also 
served  to  further  confirm  several  of  the  ideas  originally  generated  by  members  of  the  VESTED 
integrated  project  team;  as  well  as  to  illuminate  insight  about  possible  downstream  functionality. 

The  Critical  Decision  Method  (CDM),  grounded  in  a  multiple-pass  event  retrospection 
process,  was  chosen  as  the  approach  of  choice  to  guide  the  first  round  of  semi-structured 
interviews.  CDM  extends  the  critical  incident  interview  technique  by  including  opportunistic 
questions  that  elicit  aspects  of  expertise,  such  as  the  basis  for  making  perceptual  discriminations, 
conceptual  discriminations,  judgments,  and  decisions  in  a  contextually  rich  cue  stream 
(Hoffman,  Crandall,  and  Shadbolt,  1998;  Klein,  Calderwood,  and  MacGregor,  1989).  CDM 
yields  situation  records,  timelines,  and  descriptions  of  decision  requirements.  The  effectiveness 
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of  this  approach  has  been  demonstrated  in  several  domains  and  thus  it  was  selected  to  elicit 
information  about  the  video  scoping,  design,  and  production  process. 

The  CDM  approach  began  as  an  interviewee  selected  a  set  of  project  cases  that  broadly 
defined  the  parameters  of  the  subsequent  discussion.  Next,  a  participant  was  asked  to  isolate  the 
critical  incidents  in  a  particular  project  case  that  they  felt  defined  expertise.  A  participant 
recount  of  the  incident  was  elicited,  with  equal  emphasis  on  the  incident  as  well  as  on  its 
antecedents  and  outcomes.  During  the  CDM-based  dialogue,  the  interviewer  diagramed  a 
timeline  of  events  and  the  basic  stage-specific  incident  information  (e.g.,  stakeholders,  facts, 
resources,  etc.).  The  mapped  timeline  was  then  shared  with  the  interviewee  for  verification 
and/or  revision.  After  common  ground  was  established  about  the  people,  systems,  and  data  in 
play  in  a  project,  the  CDM  process  proceeded  via  progressive  deepening  to  further  distill  lessons 
learned  that  would  inform  VESTED’s  functionality. 

The  second  round  of  interviews  was  anchored  in  a  scenario-based  process.  Interviewees 
were  asked  to  participate  by  role-playing  a  character  that  owned  a  video  production  studio  in  a 
single  video  production  scenario  that  consisted  of  four  interrelated  sub-events.  The  four  events 
presented  conditions,  issues  and  core  challenges  that  spanned  project  scoping  to  project 
completion  during  a  fictitious  but  realistic  video  production  project  with  multiple  stakeholders. 

A  total  of  three  scenario-based  interviews  were  conducted. 

The  third  round  of  interviews  was  characterized  by  an  open,  unstructured  dialogue  with 
two  SMEs  with  extensive  expertise  in  storyboarding  and  animated  storyboarding.  The 
conversations  which  ensued  were  centered  on  the  strengths  and  limitations  of  five  commercially 
available  and  widely-used  storyboarding  tools  (e.g.,  Celtx,  Final  Draft,  FrameForge,  Hollywood 
Screenwriter,  Toon  BOOM). 

The  fourth-,  and  final-round,  of  interviews  involved  presenting  screenshots  of  the  present 
implementation  of  VESTED  to  three  SMEs  in  order  to  gather  feedback  about  anticipated  tool 
content  and  functionality;  as  well  as  about  the  proper  placement  and  sequencing  of  VESTED’s 
features.  SME’s  affective,  utility  and  difficulty  reactions  to  VESTED  were  also  operationalized. 

In  order  to  better  inform  the  VESTED  design  process,  the  insight  generated  during  each 
phase  of  the  interview  process  about  video  project  scoping,  planning  and  production  was  mined, 
vetted  in  meetings  by  the  research  team,  and  where  appropriate,  leveraged  to  enhance  VESTED 
functionality.  For  example,  several  SMEs  lamented  the  lack  of  actionable  advice  about  how  to 
accurately  estimate  the  total  costs  associated  with  planning,  executing  and  delivering  a 
multimedia  project  to  their  clients.  Although  it  was  the  original  intent  of  the  research  team  to 
address  cost  and  benefit  issues  via  VESTED’s  features,  the  expert  feedback  from  these  potential 
end-users  ultimately  served  to  increase  our  confidence  that  this  was  indeed  a  worthwhile  pursuit 
amongst  the  many  competing  functionality  options  for  limited  project  resources. 

Design  guidance  gathered  from  the  SME  interviewees  that  could  not  be  fully 
incorporated  into  the  current  version  of  VESTED  was  assembled  into  a  master  list  of  features 
appearing  below.  SME  interviewees  suggested  a  wide-range  of  controls,  features  and  functions 
that  would  potentially  further  enhance  the  usability  and  effectiveness  of  VESTED,  including:  (1) 
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pre-production  cost-benefit  and  inventory  tracking  tools,  (2)  drawing  tools,  (3)  editing  tools,  (4) 
production  tools,  (5)  tools  for  creating  animatics,  and  (6)  pipeline  integration  tools.  The  features 
and  functionality  culled  from  the  interview  process  which  were  deemed  most  pertinent  to 
improving  VESTED’s  design  in  the  future  are  listed  below. 

•  Auto-populating  Timeline 

•  Complete  Project  Timeline  View 

•  Project  Time  Calculator 

•  Equipment  Tracking  Tool 

•  Production  Tracking  Form 

•  Shot  List  Tool 

•  Shooting  Schedule  Form 

•  Pre-visualization  of  Photo  Accurate  Set 

•  Multi-camera  Control  Room  Interface 

•  Object  Importing  and  Authoring 

•  Object  Relationship  Assistance 

•  Character  Expression  Manipulation 

•  Correlated  Film  Media  and  Aspect  Ratio 

•  Correlated  Film  Size  and  Aperture 

•  Editable  Set  Parameters 

•  Embedded  Room  Building  Assistance 

•  Snap  Construction  of  Objects  and  Environments 

•  Object  to  Object  Smoothing 

•  Keyword-based  Search  of  Object  Libraries 

•  First-person  and  Overhead  Diagramming 

•  First-person  and  Overhead  Blueprinting 

•  Camera,  Crane  and  Boom  Manipulation 

•  Camera  Meta-data  Tagging  and  Printing 

•  Pixel  or  Multi-layer  Vector  Manipulation 

•  Flip  Book  and  Onion  Skinning 

•  Definable  Panel  and  Layer  Attributes 

•  Auto-generated  Storyboards  from  Images 

•  Real-time  Storyboard  Preview 

•  Script  Creation  and  Importing 

•  Auto-conversion  of  Scripts  to  Comply  with  Standards 

•  Import  of  Soundtracks  and  Volume  Editing 

•  Export  of  Storyboards  in  .mov  and  .swf  Formats 

•  Export  of  Storyboards  to  Photoshop  via  .psd  Files 

In  addition  to  the  SME  dialogues  described  above,  interviewees’  reactions  to  the 
VESTED  tool,  given  the  envisioned  condition  that  it  was  fully  instantiated  in  working  software 
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application  form,  were  also  captured  during  the  fourth-  and  final-round  of  interviews.  SME’s 
affective,  utility  and  difficulty  reactions  to  VESTED  were  operationalized  via  a  verbal 
administration  of  three  standardized  reactions  measures.  The  findings  of  this  process  provided 
quantitative  evidence  for  the  theorized  proximal  mediating  variables  (viz.,  positive  instructor 
affective  and  utility  reactions)  of  the  effects  attributable  to  VESTED,  as  depicted  in  Figure  1. 

Interviewee/instructor  respondents  reported  favorable  affective  reactions  (16  out  of  a 
possible  21)  and  utility  reactions  (16  out  of  a  possible  21)  to  VESTED.  These  findings,  while 
very  preliminary,  suggest  that  interviewees/instructors  perceived  that  VESTED  would  be  both 
enjoyable  to  use,  and  effective  for  creating  demonstrations  for  training,  should  it  ultimately  be 
fully  implemented  in  software  form.  Of  note,  respondents  also  reported  that,  sans  a  robust 
embedded  help  menu  system,  VESTED  may  be  somewhat  difficult  to  leam  to  use  and  apply  to 
create  demonstrations  for  training  (9  out  of  a  possible  14  in  terms  of  difficulty  reactions). 
Collectively,  these  findings  suggest  VESTED  would  be  well  received  by  end-users  as  well  as 
valuable  as  a  decision  support  tool  during  the  demonstration  specification  and  design  process. 
The  findings  also  suggest  that  additional  embedded  guidance  and  support  is  warranted  in  order  to 
ensure  VESTED  users  can  maximize  the  value  of  this  tool  in  support  of  their  training  efforts. 


Formulation  of  Guidelines  for  Demonstrations 
Why  Not  To  Demonstrate 

Before  beginning  to  fonnulate  guidelines  for  constructing  training  demonstrations,  it  is 
important  to  consider  the  prerequisite  questions  of  whether  any  implementation  or  use  of  (new) 
training  demonstrations  are  needed  or  warranted  in  the  application  domain  of  interest. 
Demonstrations  are  not  necessarily  always  used  in  training  for  a  variety  of  reasons,  sometimes 
justifiably  and  maybe  sometimes  not.  A  Wall  Street  Journal  article  (Pasztor,  2009)  describes  a 
National  Transportation  Safety  Board’s  investigation  of  an  airliner  crash  in  which  lack  of 
demonstration  of  a  key  procedure  may  have  been  implicated  as  a  cause.  Accordingly,  in  addition 
to  addressing  the  positive  reasons  why  a  demonstration  may  be  warranted  in  a  training  program 
using  an  objective-driven  design  approach,  it  is  useful  also  to  consider  the  reasons  why 
demonstrations  may  be  rejected.  Some  of  the  principal  reasons  for  rejection  are: 

•  Production  of  a  demonstration  that  would  be  good  enough  to  be  beneficial  would  be 
too  costly  (in  terms  of  financial  budget,  time,  human  resources,  etc.). 

•  Production  of  a  demonstration  would  be  too  dangerous  (e.g.,  for  showing  how  to 
recover  from  crisis  situations). 

•  A  demonstration  would  focus  the  trainee  on  a  single  case  or  a  very  small  number  of 
cases  rather  than  the  development  of  general  abilities  that  can  readily  be  adapted  to 
diverse  conditions;  therefore,  a  focus  on  individual  cases  will  tend  to  take  the  focus 
away  from  the  needed  abstraction  and  generalization. 

•  The  task  is  performed  very  differently  by  different  experts  or  in  slightly  different 
conditions  so  it  may  be  misleading  to  demonstrate  just  one  or  a  few  of  these 
examples. 

•  The  relevant  behaviors  are  primarily  cognitive  and,  therefore,  cannot  normally  be 
visualized-there  is  little  or  nothing  that  is  normally  observable  in  the  target  behaviors. 
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•  The  existing  training  program  has  always  worked  just  fine  without  any 
demonstrations  so  there  is  no  reason  to  add  any  now. 

•  It  is  very  difficult  to  see  and  understand  the  subtle  behaviors  that  are  critical  to 
successful  performance  and  it  is  likely  that  trainees  would  be  distracted  by  the  many 
more  salient  but  irrelevant  aspects  of  behavior. 

•  Demonstrations  are  viewed  passively  by  trainees  whereas  active  engagement  in  the 
training  process  is  critical  for  successful  training-some  or  many  trainees  may  not  pay 
close  attention  or  leam  very  effectively  from  passive  observation  of  a  demonstration. 


It  is  crucial  for  the  design  process  to  address  all  of  these  concerns  to  the  degree  that  they 
are  applicable.  Still,  we  assume  that,  in  general,  the  instructional  developer  will  want  to  at  least 
explore  the  possibility  of  producing  a  new  or  improved  demonstration  in  order  to  benefit  a 
training  program. 


Key  Questions  to  Drive  Design  Guidance  and  Initial  Guidance 

The  basic  idea  for  VESTED  design  guidance  is  to  assist  the  user  in  specifying  the 
objectives  of  the  training  program  (thus  addressing  the  scope  of  the  total  training  program,  not 
just  the  demonstration)  and  then  in  elaborating  on  those  objectives  to  the  point  where  general 
heuristic  rules  (that  we  develop  as  much  as  possible  from  archival  research  publications)  can  be 
used  to  generate  suggestions  and  specifications  for  scripts  for  training  components,  specifically 
for  training  demonstrations. 

Demonstration  design  guidance  can  be  structured  in  the  form  of  a  process  flow  diagram 
consisting  of  a  network  of  questions  with  branching  to  general  and  detailed  recommendation 
nodes  determined  according  to  the  answers  to  the  questions.  The  questions  are  concerned  with 
the  various  relevant  aspects  of  training  objectives,  target  task  characteristics,  and  target  trainee 
population  characteristics.  Following  are  some  of  the  key  questions  for  this  process  flow 
followed  by  some  initial  suggestions  for  guidance  that  would  be  provided  for  each  of  the 
possible  answers  to  the  questions: 

a.  To  what  degree  is  the  focal  task  comprised  of  potentially  observable  physical  action 
versus  unobservable  cognitive  activity? 

(1)  All  observable  physical  action-Plan  on  standard  video  demonstration  of  complete 

task. 

(2)  Partly  physical  action  and  partly  cognitive-Plan  on  video  demonstrations  of 
observable  task  components  with  verbal  explanations  and/or  cognitive  modeling  (i.e.,  “thinking 
aloud”)  to  represent  cognitive  aspects. 

(3)  All  cognitive-Plan  on  primarily  verbal  explanation  of  task  augmented  with  visual 
and  auditory  illustrations  of  key  principles  if  possible. 
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b.  Is  the  observable  physical  action  (especially  the  most  challenging  aspects  to  be 
learned)  primarily  dynamic  movement  or  static  posture? 

(1)  Dynamic  movement-Plan  on  standard  video  demonstration  focusing  especially  on 
the  most  difficult  aspects  of  the  required  movements,  such  as  by  zooming  in,  slowing  down,  etc. 

(2)  Static  postures-Plan  on  video  demonstration  of  postures,  using  techniques  such  as 
panning  and  highlighting  to  draw  attention  to  the  most  difficult  aspects  of  the  postures  and  also 
providing  clear  illustration  of  how  the  postures  are  sequenced  and  transitioned  along  with  any 
associated  required  movements. 

c.  Are  <some/many/all>  of  the  key  aspects  of  task  performance  typically  observable 
from  <one/multiple>  visual  vantage  points? 

(1)  Yes-Plan  on  standard  video  demonstration  of  complete  task  as  viewed  from 
appropriate  perspectives.  Also,  plan  to  employ  key  perspectives  that  have  special  importance, 
such  as  the  perspective  of  the  adversary,  perspectives  that  reveal  special  risks/vulnerabilities,  and 
perspectives  that  emphasize  other  special  task  challenges. 

(2)  No-Try  to  create  VE  perspectives  for  effective  observation  of  key  task  actions 
through  image  manipulations  such  as  creation  of  transparencies,  fish-eye  magnification,  etc. 

d.  Identify  the  most  important  component  lessons  or  visual  pattern  distinctions  that  are 
readily  rendered  as  visual  (or  visual  and  verbal/auditory)  examples  or  illustrations.  Plan  to  create 
demonstration  vignettes  for  each  component  lesson  or  important  visual  pattern.  Limit  the 
number  of  such  lesson  vignettes  in  order  to  keep  the  total  demonstration  to  a  manageable  length 
for  the  trainee. 

e.  Are  any  of  the  critical  visual  elements  subtle  or  otherwise  difficult  to  see  and/or 
understand  at  normal  speed  and  with  nonnal  perspective  and  without  elaborating  explanation? 

(1)  Yes-Plan  to  use  highlighting,  zooming/magnification,  multiple  perspectives, 
verbal  annotation,  etc.  to  draw  attention  and  explain  features  and  issues. 

(2)  No-Check  again  by  interviewing  trainees  who  have  the  most  difficulty  with  initial 
standard  demonstrations  of  the  task. 

f.  Are  there  any  especially  prominent  (and  relatively  likely  and  performance-critical) 
errors  in  task  perfonnance  that  deserve  to  be  explicitly  addressed?  Do  standard  tutorial 
presentations  specifically  identify  certain  errors  and  instruct  the  trainee  as  to  why  and  how  to 
avoid  them? 

(1)  Yes-Plan  to  include  component  demonstrations  of  no  more  than  a  few  of  the  most 
common  and  critical  errors,  using  visual  and  auditory  elements  to  make  it  clear  that  these  are 
errors  to  be  avoided  and  not  to  be  confused  with  recommended  performance.  For  each  case, 
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provide  a  clear  explanation  of  what  is  wrong  with  the  erroneous  perfonnance,  why  it  is  wrong 
(i.e.,  consequences),  and  how  to  avoid  making  this  error.  Before  and  after  presenting  error 
demonstrations,  make  sure  to  present  correct  perfonnance  demonstrations  so  as  to  ensure  that  the 
trainee  especially  leams  and  remembers  the  correct  task  performance  and  avoids  any  possibility 
of  confusing  incorrect  with  correct  performance.  Make  sure  to  provide  at  least  equal  time  to  the 
demonstration  of  correct  performance  relative  to  all  error  demonstrations  to  ensure  that  correct 
performance  is  firmly  fixed  in  the  trainee’s  memory,  with  no  chance  that  the  trainee  might 
confuse  any  presentations  of  errors  with  correct  perfonnance. 

(2)  No-If  the  pattern  of  errors  is  too  diverse  with  each  specific  error  being  relatively 
unlikely,  then  avoid  presentation  of  specific  error  demonstrations  because  they  are  unlikely  to  be 
very  helpful  and  the  downside  risk  of  later  confusion  with  prescribed  performance  would  be  too 
much  of  a  concern. 

g.  Do  significant  aspects  of  task  performance  need  to  be  altered  to  contend  with  typical 
variations  in  environment  and  contextual  conditions?  Are  there  multiple  distinct  key 
lessons/principles  to  be  learned  in  association  with  distinct  environment/context  conditions? 
(Must  determine  to  what  degree  the  relevant  lessons/principles  can/should  be  learned  through 
verbal  explanation  versus  demonstration  examples  for  each  case,  though  combinations  of  the  two 
are  likely  to  be  especially  effective.) 

(1)  Yes-Identify  a  few  distinct  environment/context  condition  cases  that  effectively 
bracket  and  represent  the  key  variations  of  concern.  Plan  on  separate  demonstrations  or 
demonstration  variations  for  each,  in  each  case  highlighting  the  key  differences  between  the 
cases  and  how  the  trainee  should  be  able  to  identify  which  case  is  most  relevant  to  any  situation 
that  arises. 


(2)  No- At  least  at  the  current  stage  of  training,  variations  in  environment  and  context 
are  not  significant  concerns.  A  single  demonstration  of  recommended  performance  within  a 
nominal  context  is  sufficient. 

h.  Are  there  significant  variations  in  correct  task  performance,  with  distinct  variations 
being  more  or  less  conducive  to  the  capabilities  and  limitations  of  individual  trainees? 

(1)  Yes-Identify  a  few  of  the  primary  alternatives  in  style  or  procedure  for  acceptable 
task  performance  and  develop  explanations  for  how  trainees  should  determine  which  is  most 
appropriate  for  them  to  adopt  individually.  Plan  on  separate  demonstrations  for  each  of  the 
primary  styles/procedures,  in  each  case  highlighting  the  key  differences  and  how  the  trainee 
should  be  able  to  identify  which  is  most  suitable. 

(2)  No- At  least  at  the  current  stage  of  training,  variations  in  procedure  and  style  are 
not  of  concern.  A  single  demonstration  with  a  single  procedure  and  style  of  task  performance  is 
all  that  is  warranted. 
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i.  With  respect  to  each  area  of  identified  or  expected  specific  task  performance  difficulty, 
is  the  difficulty  mainly  due  to: 

(1)  Difficulty  in  identifying  the  triggering  conditions  for  special  action  (including 
cognitive  action  as  well  as  physical  action)?  Plan  to  develop  segments  of  the  demonstration  to 
focus  on  how  to  recognize  the  triggering  conditions  and  how  to  avoid  the  most  common  errors  in 
recognition. 

(2)  Difficulty  in  remembering  to  attend  to  the  situational  indicators  and  then  perfonn 
the  appropriate  action?  Plan  on  providing  repetitive  emphasis  on  the  need  for  vigilance  in  this 
area  along  with  explanation  of  why  it  is  important  and  possibly  also  offering  detailed  guidance  as 
to  how  to  manage  the  attention  process  (e.g.,  such  as  the  instrument  scan  patterns  that  aircraft 
pilots  are  taught). 

(3)  Difficulty  in  understanding  how  to  perfonn  the  appropriate  action  correctly? 
Identify  the  areas  of  significant  difficulty  in  terms  of  cognitive  and  physical  actions.  Problems  in 
cognitive  areas  should  be  addressed  through  explanation  of  required  cognitive  processes  (what, 
why,  how).  Potentially  problematic  subsidiary  processes  should  also  be  considered  for  attention, 
especially  if  a  significant  portion  of  the  trainees  are  expected  to  have  difficulty  with  the 
subsidiary  processes.  Problems  in  physical  aspects  of  performance  should  be  addressed  through 
explicit  demonstration  of  correct  perfonnance  with  highlighting  of  performance  in  the  areas  of 
specific  difficulty.  Demonstration  of  frequent  errors  should  be  considered  in  accordance  with 
Guideline  6. 

(4)  Difficulty  in  actually  performing  the  required  action  correctly  despite  knowing 
when  and  how  to  do  it?  This  difficulty  cannot  be  further  aided  via  demonstration  but  rather 
requires  interactive  experiential  training  with  feedback  to  achieve  improvement. 

j.  Most  training  will  be  concerned  with  complex  skills  that  will  be  composed  of 
integrated  combinations  of  simpler  skills  to  accomplish  an  objective.  In  demonstrating  the 
complex  skills,  we  need  to  consider  to  what  degree  we  should  assume  that  the  trainees  have 
mastered  the  simpler  component  skills  and,  hence,  to  what  degree  the  training  could  benefit  from 
remediative  demonstration  of  those  simpler  skills. 

(1)  Potentially  problematic  component  subskills/subtasks  can  be  identified  (e.g., 
through  observation,  analysis,  evaluation  of  performance  or  interviews  with  trainees).  Plan  to 
develop  concise  segments  of  review/remediation  for  subskills/sub  tasks  of  concern.  Try  to  distill 
review  pieces  to  minimal  length  by  employing  cues  to  salient  elements  of  earlier  training  on 
those  subskills/subtasks.  Alternatively,  plan  to  provide  elements  in  periodic  evaluations  of 
trainee  perfonnance  to  identify  any  deficiencies  in  relevant  subskills/sub  tasks  and  then  offer 
remedial  demonstrations  only  upon  identification  of  the  need. 

(2)  Even  if  there  is  no  initial  indication  of  any  specific  subskill/subtask  perfonnance 
deficiencies  in  the  target  population  at  the  intended  stage  of  training,  plan  to  incorporate 
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elements  in  periodic  trainee  performance  evaluations  to  determine  if  such  problems  might 
eventually  arise,  then  would  be  addressed  as  described  in  (a)  above. 

k.  Are  the  focal  tasks  primarily  concerned  with  the  gross  physical  interactions  of 
multiple  people,  in  the  finer  physical  actions  of  individuals,  or  in  verbal  and  gestural 
communications  between  individuals? 

(1)  Gross  physical  interactions  of  multiple  people-Plan  to  use  a  tool  like  VBS2  for 
video  generation. 

(2)  Fine  physical  actions  of  individuals-Plan  to  use  a  tool  like  Gamebryo. 

(3)  Verbal  and  gestural  communications  between  individuals-Plan  to  use  a  tool  like 
VCommunic  ator . 


Vision  for  Overall  VESTED  Authoring  Use  Model 

The  VESTED  Demonstration  Specification  application  enables  users  to  author 
demonstration-based  training  videos  through  a  combination  of  training  techniques  used  in 
conjunction  with  a  storyboard  creation  tool.  Additionally,  VESTED  users  will  be  able  to  utilize 
the  multi-player  functionality  of  the  underlying  system  game-engine  in  order  to  create  complete 
instructional  demonstration  videos.  Traditionally,  this  type  of  work  was  the  domain  of  high-end 
animation  and  video  packages,  such  as  3DStudio  Max®  and  Final  Cut  Pro.  These  packages 
required  a  great  deal  of  skill  from  their  end  users  and  frequently  had  steep  learning  curves  before 
adequate  results  could  be  obtained. 

Before  continuing,  it  is  important  to  articulate  the  demonstration  authoring  use-model 
envisioned  for  VESTED.  Figure  4  below  depicts  a  canonical  use-model  for  VESTED 
demonstration  authoring  comprised  of  three  roughly  sequential  steps: 

a.  Demonstration  specification  step  where  the  author  identifies  the  specific  learning 
objectives  to  be  modeled,  reviews  demonstration  authoring  guidelines,  and  creates  a  set  of 
demonstration  storyboards  that  reflect  the  target  behaviors. 

b.  Demonstration  video  generation  step  whereby  the  demonstration  author  utilizes  the 
specific  game -based  environment  to  instantiate  the  behaviors  represented  in  the  storyboards  (thus 
generating  a  set  of  raw  intermediate  videos). 

c.  Video  mixing  step,  which  involves  the  importation  of  those  intermediate  videos  into  a 
commercial-off-the-shelf  video  mixing  tool  whereby  post-processing  can  be  performed  to  create 
the  final  demonstration  videos. 
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Figure  4.  VESTED  authoring  use-model. 

We  anticipate  that  the  demonstration  pipeline  for  VESTED  demonstration  creation  will 
involve  a  number  of  users  within  the  use-model  described  above.  First,  we  anticipate  that  the 
articulation  of  learning  goals  and  the  creation  of  demonstration  storyboards  during  the 
demonstration  specification  step  will  typically  be  performed  by  a  domain  subject  matter  expert 
(SME)  well-versed  in  the  training  materials  being  demonstrated.  Second,  we  anticipate  that  the 
demonstration  author,  most  likely  an  engineer,  will  typically  work  closely  with  the 
demonstration  author  in  order  to  instantiate  the  demonstration  using  the  authoring  toolset  within 
a  given  game-engine  in  order  to  generate  a  body  of  raw  intermediate  videos.  Digital  animators 
may  also  be  required  during  this  step  if  a  given  game-engine  does  not  provide  a  required 
animation.  Lastly,  we  envision  that  the  creation  and  compositing  of  a  final  demonstration  video 
will  be  done  by  a  video-editor  working  closely  with  the  SME  who  created  the  demonstration 
storyboard  in  order  to  ensure  that  the  desired  instructional  aspects  of  the  demonstration  are 
reflected  in  the  final  demonstration  mixing  process. 


Demonstration  Specification  -  Design  and  Storyboarding 


Demonstration  Specification  is  the  VESTED  process  whereby  a  demonstration  author 
selects  the  specific  learning  goals  to  be  demonstrated  and  then  elaborates  those  requirements  into 
a  detailed  design  for  the  demonstration.  Based  on  selected  learning  goals,  the  VESTED 
authoring  tool  compares  those  selected  goals  to  the  underlying  demonstration  guidelines  to 
provide  recommendations  about  content  sequencing,  choice  of  instructional  strategies  (e.g., 
comparisons  and  contrasts),  level  of  fidelity,  and  so  forth.  Design  decisions  are  represented  in 
the  form  of  demonstration  design  hypotheses  (i.e.,  cases),  each  of  which  is  linked  to  a  rationale 
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and  to  sources  of  data.  During  the  subsequent  storyboard  stage,  the  author  will  be  able  to  define 
high-level  demonstration  scenes  and  how  they  are  to  be  ordered  and  sequenced. 

For  example,  an  author  may  want  to  construct  a  demonstration  that  focuses  on  the  MOUT 
domain  in  the  instructional  context  of  providing  a  building-clearing  scenario,  but  may  also  want 
to  focus  on  the  role  of  an  individual  within  that  team  context.  To  do  so,  the  VESTED  author 
would  select  a  MOUT  domain  template.  This  template  would  configure  the  authoring  tool  to 
visually  highlight  the  specific  VESTED  functions  most  relevant  to  the  authoring  process  for  a 
MOUT  demonstration,  such  as  pertinent  visual  scene  props,  use  of  characters  if  applicable, 
details  of  blue/red/neutral  forces,  character  movement,  object  behaviors,  and  use  of  mixed  media 
(such  as  imported  videos).  Additionally,  by  specifying  the  “individual  vs.  team”  template,  the 
VESTED  authoring  tool  provides  access  to  tailored  guidelines  relative  to  this  form  of 
demonstration.  When  the  research  literature  warrants  it,  these  guidelines  could  be  expanded  to 
address  what  perspectives  to  include  in  demonstration  displays,  whether  to  use  demonstrations  of 
common  errors,  whether  and  how  to  employ  distortions  in  time  scale,  and  how  to  handle  decision 
points  within  the  task  being  modeled.  In  addition  to  these  forms  of  guideline  support,  we 
anticipate  including  a  library  of  hierarchically  organized  hypertext  reference  documents 
(typically  field  manuals)  to  provide  rapid  access  to  doctrinal  aspects  of  each  supported  domain 
template  (e.g.,  MOUT,  cultural  familiarization,  etc.)  and  to  allow  demonstration  authors  to 
include  linkages  between  doctrinal  references  and  authored  demonstrations  for  pedigreed 
VESTED  demonstrations.  A  formative  set  of  the  training  objective  templates  include  the 
following: 

•  Individual  vs.  team-In  some  cases,  the  objective  will  be  to  demonstrate  just  the 
task  performance  of  a  single  individual  (even  though  that  person  may  be  part  of  a 
team  in  their  broader  responsibilities),  and  in  other  cases  the  objective  may  be  to 
demonstrate  how  to  coordinate  individual  perfonnance  with  the  perfonnance  of 
other  team  members. 

•  Gestalt  vs.  detail-The  objective  may  be  to  present  the  complete  integrated  gestalt 
representation  of  the  focal  task  or  activity,  or  it  may  be  primarily  just  to  focus  on 
a  few  detailed  aspects  of  the  activity  on  the  assumption  that  overall  gestalt  is 
already  well-enough  understood  by  the  target  audience.  Although  some  complete 
integrating  demonstration  framework  may  still  be  warranted  in  many  of  the  cases 
where  the  main  focus  is  on  detail,  the  detailed  objective  may  warrant  a  very 
coarse  representation  outside  the  vicinity  of  the  focal  details.  Where  details  are  to 
be  the  focus,  some  of  the  main  options  will  likely  include:  first-person  physical  or 
speech  behaviors,  third-person  physical  or  speech  behaviors,  external 
visual/speech/sound  cues,  and  abstract  concepts  (e.g.,  line  of  sight,  field  of  view, 
response  time). 

•  Nominal  vs.  adaptive-The  objective  may  be  to  convey  the  image  and  basic 
understanding  of  a  single,  nominal,  correct  way  to  perfonn  the  task,  or 
alternatively  to  try  to  convey  a  deeper  understanding  of  the  reasoning  why  that 
procedure  is  generally  appropriate  and  thereby  to  instill  a  capability  to  adapt  the 
procedure  effectively  when  the  situation  does  not  evolve  according  to  the 
assumptions  of  the  nominal  scenario. 
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•  Correct  performance  vs.  errors-We  assume  that  in  all  cases  we  will  want  to 

provide  a  demonstration  of  correct  performance  and  in  some  cases  that  may  be  all 
that  is  warranted.  But  in  other  cases,  it  will  also  be  important  to  present 
demonstrations  of  common  errors,  especially  where  the  primary  focus  of  training 
is  to  avoid  common  mistakes  (such  as  in  training  of  safety  procedures  or  in 
cultural  sensitivity  where  the  main  focus  is  often  on  identifying  what  not  to  do). 


Demonstration  Video  Generation  -Intennediate  Media  Generation 


Demonstration  Video  Generation  is  the  VESTED  process  whereby  a  demonstration 
author  utilizes  a  set  of  environment  assets  (pre-supplied  ‘whole’  environments  or  ‘library-based’ 
assets),  places  actors  (blue/red-forces),  defines  behaviors  (including  avatar  movement  and 
positioning),  iteratively  reviews  demonstrations  and  refines,  and  generates  raw  source  videos 
(presumably  from  multiple  camera  perspectives).  A  range  of  VESTED  authoring  concepts  were 
identified  and  are  discussed  below. 

Sandboxed  environments  contain  world  geometry  and  enemy  AI  already  positioned  in  the 
environment.  The  author’s  control  is  limited  to  removing  or  modifying  the  behavior  of  the  pre¬ 
placed  enemy  AI,  changing  camera  views,  and  adding  simple  events  (e.g.,  audio  events  like  gun¬ 
fire,  etc.).  Existing  AI  middleware  solutions  often  require  heavy  markup  of  the  environment  and 
tweaking  for  proper  behavior.  This  approach  requires  that  pre-constructed  environments  are 
provided  in  a  library  so  that  the  user  does  not  need  to  understand  the  nuances  required  to  set  up 
an  effective  environment  using  this  type  of  AI  approach.  Through  the  use  of  preconfigured 
environments,  the  users’  learning  curves  would  be  reduced  and  they  would  be  able  to  begin 
creating  demonstrations  much  more  quickly.  Use  of  preconfigured  environments  also  provides  a 
possible  content  creation  mechanism  where  updates  can  be  offered  for  the  application  to  provide 
novel  locations  and/or  enemy  behaviors  for  existing  packages. 

Demonstrations  created  through  a  multiplayer-capture-based  design  involve  recording 
the  actions  of  multiplayer  participants  coupled  with  a  playback  mechanism  from  user-selectable 
and  editable  ‘virtual’  cameras.  Setting  up  a  scenario  for  capture  is  limited  to  choosing  an 
environment  and  connecting  to  the  other  participants.  After  the  scenario  has  ended,  the  author 
reviews  the  scenario  playback,  moving  as  desired  between  placed  cameras,  and  records  video  as 
desired  from  all  instructionally  meaningful  camera  views.  Multiplayer’s  most  obvious  limitation 
is  that  it  potentially  requires  a  large  number  of  skilled  participants  to  create  an  effective 
demonstration.  However,  the  removal  of  any  AI  entities  from  this  design  helps  reduce  the  cost 
of  development. 

Demonstrations  created  through  ghosting  are  similar  to  the  capture  through  the 
multiplayer  approach  with  the  exception  that  a  single  user  plays  all  parts  of  the  scenario.  To 
accomplish  this,  an  author  would  move  through  the  environment  once  for  each  actor  in  the 
scenario.  As  the  author  adds  each  new  character  to  the  scenario,  the  previous  runs  are  visible  as 
ghosts  in  the  environment.  Using  this  layering  approach,  a  single  user  can  build  up  a  complex 
scenario  over  time.  Once  the  author  is  satisfied  with  the  composition  of  the  scenario,  it  can  be 
played  back  and  the  results  loaded  into  the  same  non-linear  editing  interface  used  in  the 
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multiplayer  capture  approach.  Ghosting  places  a  high  cognitive  load  on  the  author  and  is  poorly 
suited  for  large  scale  simulations  containing  many  entities. 

Custom  demonstrations  require  the  author  to  manually  place  all  content,  using  both  an 
interactive  3D  environment  coupled  with  script  actions  using  a  timeline/track  interface.  This 
approach  can  be  mixed  with  limited  AI  behaviors  to  help  ease  the  burden  on  the  author  (e.g., 
building  a  script  behavior  to  ‘sit’  that  positions  the  character  correctly  next  to  a  chair  object  and 
correctly  plays  the  necessary  animations  required  to  add  realism  to  the  invocation  of  the 
command).  This  paradigm  gives  the  most  control  of  all  of  the  approaches,  but  the  tradeoff  is  that 
the  scenario  creation  process  itself  is  highly  iterative  and  requires  ‘trial-and-error’  as  behavior 
scripts  can  interact  in  ways  that  lead  to  unpredictable  behaviors  -  thus  requiring  considerable 
refinement. 


Demonstration  Mixing  -  Instructional  Integration  and  Final  Production 


Demonstration  Video  Mixing  is  the  VESTED  process  whereby  a  user  imports  source 
videos  (from  above),  selects  specific  source  video  ‘tracks’  (i.e.,  ‘views’),  includes  2D  overlays 
and  animations,  records  instructional  narrations,  composites  other  training  effects  (e.g.,  flash 
lighting),  and  generates  final  video  renders.  The  figure  below  illustrates  a  high-level  set  of 
training-constructs  based  on  analysis  of  existing  MOUT  video  demonstrations.  The  goal  was  to 
identify  a  small  set  of  demonstration  constructs  and  map  those  to  training  intent.  Five  specific 
construct  types  were  derived  from  the  video  and  mapped  to  corresponding  intent  types  (by  way 
of  example)  and  are  visually  depicted  in  Figure  5. 

The  description  of  each  construct  and  training  intent  include: 

•  Enumeration  -provide  a  2D  text/graphic  overlay  of  concepts  with  a  particular 
emphasis  on  knowledge-priming. 

•  Conceptualization  -provide  a  2D  or  simplified  representation  of  demonstration 
tasks  with  a  particular  emphasis  on  knowledge-priming. 

•  Modeling  -  provide  lst/3ld-person  point-of-view  presentation  of  tasks/procedures 
being  modeled;  this  has  been  the  heretofore  focus  of  the  VESTED  demonstration 
authoring  concept  of  operations  which  includes  core  enabling  components  such  as 
a  3D  environment  coupled  with  an  Al-based  authoring  feature-set. 

•  Composited  -  provide  additional  cues  to  the  modeled  behaviors  such  as  a  2D 
overlay. 

•  Focus  cuing  -  use  of  visual  and  auditory  cues,  such  as  flash-lighting,  high¬ 
lighting,  and  narration,  to  draw  attention  to  salient  cues  and  concepts  within  the 
modeled  task. 
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Figure  5.  VESTED  instructional  content  type  examples. 


The  Video  Creation  steps  described  in  the  candidate  use-model  above  are  premised  on  a 
game -based  3D  environment.  It  is  likely  that  for  VESTED  users,  the  Video  Mixing  steps 
outlined  above  may  best  be  done  using  third-party  commercial  tools  for  non-linear  video-editing 
(e.g.,  Adobe  Premiere  Elements,  Sony  Vegas,  Roxio  Creator,  Nero  9,  Pinnacle  Studio). 
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Test  Case  Applications 


Test  Case  Selection  Criteria 


In  the  course  of  VESTED  development  efforts,  we  found  that  it  was  essential  to  select 
some  specific  application  cases  in  order  to  focus  our  design  efforts.  There  is  a  broad  range  of 
procedural  learning  areas  in  which  demonstrations  are  potentially  beneficially  applicable,  with 
these  areas  of  relevant  applicability  varying  over  several  key  dimensions.  Important  differences 
in  these  areas  include  particularly: 

•  Whether  the  task  is  primarily  physical/psychophysical  or  primarily  cognitive; 

•  Whether  the  task  is  strictly  linear  (always  the  same  steps  in  the  same  order)  or 
characterized  by  multiple  conditional  branching; 

•  Whether  the  task  is  performed  by  a  solitary  individual  or  as  an  interactive  activity 
involving  cooperation  or  competition  with  other  people; 

•  The  kinds  and  amounts  of  declarative  knowledge  required  to  learn  the  task; 

•  The  kinds  and  amounts  of  recognitional  knowledge  required  to  learn  the  task. 

We  chose  two  different  test  cases  representing  distinctly  different  configurations  over 
these  and  other  task  characteristics,  one  for  the  first  year  efforts  which  addressed  VESTED 
product  conception  and  another  for  the  second  year  efforts  which  were  concerned  primarily  with 
product  implementation.  The  first  year  test-case  that  we  selected  is  one  of  the  key  tasks  within 
the  area  of  Military  Operations  in  Urban  Terrain  —  the  task  of  finding  and  occupying  a  hasty 
firing  position.  This  task  is  primarily  physical/psychophysical,  minimally  linear  (mainly 
focusing  on  learning  under  what  conditions  you  should  use  each  different  type  of  firing  position 
and  posture),  somewhat  competitively  interactive  with  adversary  personnel,  and  with  little  need 
for  declarative  knowledge  but  considerable  need  for  recognitional  knowledge  (concerning  types 
of  categories  of  terrain,  protective  structure,  and  tactical  situation).  The  second  year  test  case  is 
concerned  instead  with  the  standard  task  for  a  battlefield  medic  of  perfonning  a 
cricothyroidotomy  (better  known  as  a  tracheotomy)  which  is  a  primarily  linear  task  that  is  both 
cognitive  and  physical  and  requires  considerable  declarative  and  recognitional  knowledge. 

Year  1  Test  Case  -  MOUT  Hasty  Firing  Position 

We  developed  a  VE  concept  demonstration  midway  through  Year  1  of  the  VESTED 
project.  The  purpose  of  creating  the  interim  demonstration  was  not  to  show  how  a  VE 
demonstration  is  created  using  an  early  stage  prototype  of  the  VESTED  system;  rather  it  was 
motivated  by  the  need  to  inform  the  development  of  the  system  and  the  authoring  guidelines.  It 
provided  the  means  by  which  we  could  walk  through  the  processes  of  demonstration 
specification,  video  generation,  and  video  mixing  to  unearth  issues  early  in  the  system 
development  life  cycle  that  would  otherwise  go  undetected  until  much  later  and  possibly  too  late. 
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A  hasty  firing  position  is  “a  position  from  which  the  Soldier  can  place  fire  upon  the  enemy 
while  using  available  cover  for  protection  from  return  fire.  The  Soldier  may  occupy  it 
voluntarily  or  [the  Soldier]  may  be  forced  to  occupy  it  due  to  enemy  fire.  In  either  case,  the 
position  lacks  preparation  before  occupation.”  (Army  Field  Manual  (FM)  3-06.1 1  and  Marine 
Corps  Warfighting  Publication  (MCWP)  3-35.3).  Hasty  firing  positions  include  firing  around 
comers  of  buildings  and  walls,  through  windows,  from  a  loophole,  from  the  peak  of  a  roof,  and 
when  there  is  no  firing  position  or  cover  available.  To  limit  the  scope  of  our  concept 
demonstration  to  a  manageable  size,  (meaning  the  ability  to  develop  the  demonstration  in  a 
relatively  short  period  of  time  yet  still  provide  meaningful  insight  into  the  demonstration 
development  process),  we  chose  to  concentrate  on  firing  around  buildings  and  walls  and  through 
windows.  Other  hasty  firing  selection  criteria  included  choosing  a  practical  example  of 
procedural  and  cognitive  knowledge  acquisition  and  the  confidence  that  the  demonstration  was 
amenable  to  depiction  in  a  virtual  environment,  initially  VBS2. 

As  we  have  previously  discussed,  demonstrations  should  be  driven  initially  by  learning 
objectives  or  goals.  So  the  first  step  in  the  development  of  the  concept  demonstration  was  the 
identification  of  learning  goals  for  hasty  firing  positions  as  well  as  the  general  principles  and  key 
factors  to  consider  from  which  the  learning  goals  were  derived.  For  hasty  firing  positions,  (FM) 
3-06.1 1  offered  two  key  factors,  protection  and  concealment,  and  nine  considerations: 

•  Make  maximum  use  of  available  cover  and  concealment. 

•  Avoid  firing  over  cover;  when  possible,  fire  around  it. 

•  Avoid  silhouetting  against  light-colored  buildings,  the  skyline,  and  so  on. 

•  Carefully  select  a  new  fighting  position  before  leaving  an  old  one. 

•  Avoid  setting  a  pattern;  fire  from  both  barricaded  and  non-barricaded  windows. 

•  Keep  exposure  time  to  a  minimum. 

•  Begin  improving  your  hasty  position  immediately  after  occupation. 

•  Use  construction  material  that  is  readily  available  in  an  urban  area. 

•  Remember  that  positions  that  provide  cover  at  ground  level  may  not  provide  cover  on 
higher  floors. 

We  derived  the  following  learning  objectives  from  these  key  factors  and  general 
principles: 

•  knowledge  of  how  to  remain  covered  while  occupying  a  hasty  firing  position; 

•  knowledge  of  how  to  remain  concealed  while  occupying  a  hasty  firing  position;  and 

•  skill  in  positioning  appendages  so  as  to  minimize  exposure  to  enemy  fire. 


The  second  step  in  the  development  process  included  the  construction  of  a  script,  or 
storyboard,  which  provided  a  storyline  that  was  broken  down  into  demonstration  segments. 
Initially,  we  did  not  have  a  storyboard  template,  thus  we  created  one.  Each  demonstration 
segment  included:  transition  into  the  segment,  the  segment’s  main  modality  (video,  text,  etc.), 
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content  frame,  camera  perspective,  animation,  narration,  music,  and  transition  out  of  the  video 
segment. 


We  populated  the  storyboards  utilizing  both  the  initial  set  of  demonstration  authoring 
guidelines  discussed  above  and  the  specified  learning  goals.  The  sequence  of  demonstration 
includes  presentations  of  both  prescribed  and  proscribed  behaviors  (i.e.,  mixed  model  as  referred 
to  in  the  BMT  literature).  The  sequence  of  the  segments  was  as  follows: 

•  Presentation  of  learning  goals 

•  Presentation  of  key  factors  and  general  principles 

•  Definition  of  hasty  firing  positions 

•  Firing  around  buildings/walls  prescribed  behavior 

•  Firing  around  buildings/walls  proscribed  behavior 

•  Firing  through  windows  prescribed  behavior 

•  Firing  through  windows  proscribed  behavior 

•  Review  of  firing  around  buildings/walls  prescribed  behavior 

•  Review  of  firing  through  windows  proscribed  behavior 

•  Review  of  learning  goals 

The  storyboards  were  then  used  by  the  VE  development  team  (i.e.,  gaming  environment 
developer  and  graphic  artist).  Other  steps  in  the  concept  demonstration  development  process 
included: 

•  Art  development 

•  Voice  narration 

•  Music  selection 

•  Video  editing 

• 

Figures  6  and  7  below  show  two  screenshots  from  the  concept  demonstration.  Figure  6 
shows  the  proper  technique  for  firing  around  a  building-the  Soldier  is  in  the  prone  position  and 
is  using  the  short  stocking  technique. 


32 


Figure  6.  Correct  firing  position  around  buildings. 

Figure  7  depicts  the  proper  firing  technique  from  inside  a  window.  The  Soldier  is 
kneeling  to  reduce  exposure  and  is  back  in  the  room  to  limit  silhouetting. 


Figure  7.  Correct  firing  position  from  inside  a  window. 
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Year  2  Test  Case  -  Cricothyroidotomy 


A  cricothyroidotomy  (CRIC)  is  a  procedure  that  is  performed  on  a  patient  who  has  an  air 
passage  that  is  blocked  above  the  cricothyroid  structure  (Adam’s  apple  region).  This  is  primarily 
a  linear  task,  though  most  of  the  major  steps  in  the  task  involve  declarative  knowledge, 
recognitional  knowledge,  and  cognitive  deliberations.  The  initial  step  of  determining  whether  or 
not  the  procedure  is  warranted  for  any  particular  patient  is  rather  challenging  (considered  the 
severity  of  the  blockage  and  the  history  of  all  other  measures  that  were  attempted  and  why  they 
failed)  and  accordingly  is  not  included  in  this  test-case;  rather  we  focus  on  how  to  perform  the 
procedure  once  the  decision  has  been  made. 

The  assumed  target  audience  for  training  in  this  test-case  consists  of  individuals  in  early 
stage  training  to  serve  as  battlefield  medics  who  have  completed  Army  basic  training  and 
satisfied  all  selection  criteria.  We  assume  that  trainees  have  been  separately  trained  or  presented 
with  demonstrations  regarding  the  following  initial  steps  of  the  CRIC  procedure: 

•  How  to  make  the  decision  of  whether  or  not  to  perfonn  the  CRIC  procedure. 

•  How  to  assemble  all  of  the  required  materials  for  the  CRIC  procedure. 

•  How  to  prepare  the  patient  for  the  CRIC  procedure. 

Following  the  viewing  of  this  demonstration  video,  trainees  are  assumed  to  receive  interactive 
training  in  the  CRIC  procedure. 

The  primary  training  objectives  to  be  addressed  by  the  demonstration  are  that  after 
viewing  the  demonstration  video,  the  trainee  should  be  able  to: 

•  List  all  steps  in  CRIC  procedure  in  correct  order 

•  Locate  anatomical  landmarks 

•  Find  appropriate  location  for  skin  incision 

•  Find  appropriate  location  for  tracheal  incision 

•  Explain  how  to  handle  most  common  problems  in  anomalous  neck  anatomy 

•  Explain  how  to  detennine  if  airway  is  patent  after  spreading  tracheal  incision 

Based  on  all  of  these  requirements,  we  formulated  the  following  demonstration  outline 
which  served  as  the  focal  material  for  development  of  all  of  the  VESTED  interface  screens  (and 
serves  as  the  example  test-case  in  all  of  the  interface  screens  depicted  in  this  report): 

(1)  Introduction 

•  Title 

•  Credits 

•  Target  audience 
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(2)  Training  objectives 

•  Assumed  prior  training 

•  Specific  training  objectives 

•  Assumed  subsequent  training 

(3)  Full  run-through  demonstration  of  basic  CRIC  procedure 

Demonstrate  standard  procedure  in  continuous,  real-time  mode  without  any  special 
problems.  Narration  should  explain  major  steps  and  highlighting  or  zooming  may  be 
used  to  draw  attention  to  focal  activities. 

(4)  Drill-down  elaboration 

■  Outline  drill-down  segments 

■  Anatomical  landmarks 

•  Basic  anatomy 

•  Problem  anatomy 

■  Skin  incision 

•  Rationale  for  vertical  incision 

•  Problems  with  excessive  bleeding 

■  Tracheal  incision 

•  Rationale  for  horizontal  incision 

•  Stabilizing  thyroid  cartilage 

•  Risk  of  esophagal  perforation 

•  Opening  incision 

■  Tube  insertion 

•  Verify  correct  insertion 

•  Verify  bilateral  breath  sounds 

•  Rectifying  if  ventilation  is  unilateral 

■  Most  common  complications 

•  Hemorrhage 

o  Causes  -  laceration  of  superficial  capillaries  or  major 
vessels 

o  Treatment  -  use  direct  pressure  or  ligation  if  necessary 

•  Esophagal  perforation 

o  Causes  —  too  deep  of  an  incision  or  forcing  ET  tube  into 
esophagus 

o  Treatment  -  surgical  repair  at  higher  echelon  of  care 

•  Subcutaneous  emphysema 

o  Causes  -  too  wide  of  an  incision  and  air  leaking  from 
insertion  site  under  skin 

o  Treatment  -  none  necessary,  will  spontaneously  resolve 
within  few  days;  placement  of  petroleum  gauze 
dressing  around  incision  site  will  help  reduce  incidence 
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(5)  Review  of  complete  procedure.  Continuous  demonstration  of  complete  procedure 
with  accompanying  narration  to  explain  special  issues  and  problems  and  review  to  avoid 
problems  and  handle  them  if  they  arise,  possibly  slowing  or  pausing  demonstration  to 
provide  opportunity  for  appropriate  amount  of  narration. 


In  order  to  better  understand  the  challenges  involved  in  making  a  VE  model  and  video  to 
accompany  this  kind  of  demonstration,  we  undertook  this  effort  in  a  limited  fashion.  Example 
views  generated  in  this  effort  are  illustrated  in  Figure  8  and  9. 


Figure  8.  View  of  cricothyroid  anatomical  structure  from  VE  model. 
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Figure  9.  Use  of  transparency  to  highlight  anatomical  structures. 
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Demonstration  Specification  Development 


Who  are  the  users? 


Of  course,  there  are  many  people  and  teams  developing  VE -based  training  products,  with 
many  successful  such  products  in  prominent  use,  such  as  in  the  case  of  “America’s  Army”. 
However,  it  is  our  suggestion  that  most  of  these  efforts  have  arisen  through  processes  other  than 
a  ‘normal’  training  product  development  process.  We  conceive  of  the  normal  development 
process  as  one  in  which  a  training  requirement  is  identified  along  with  various  relevant 
constraints  (budgets,  schedules,  success  criteria,  etc.)  followed  by  a  systematic  process  of 
formulating  alternative  strategies  to  satisfy  that  requirement  and  then  efforts  to  fonnulate  and 
execute  an  effective  implementation  plan.  Instead,  most  existing  VE  training  products  have  been 
produced  primarily  as  demonstrations  of  technological  feasibility,  vehicles  of  research 
investigations,  objects  of  R&D  programs  (such  as  the  VESTED  project  reported  here),  or  cases 
of  opportunistic  leveraging  on  VE  products  originally  produced  for  other  purposes  (such  as  for 
the  various  computer  games  that  have  been  found  to  be  adaptable  for  training  purposes; 

Belanich,  Sibley  &  Orvis,  2004). 

Thus,  we  expect  that  the  people  and  teams  who  developed  the  existing  VE  training 
products  may  be  distinctly  different  from  the  users  that  we  hope  to  support  for  the  more  nonnal 
processes  to  produce  training  development  products.  Clearly,  teams  that  have  produced 
demonstrations  of  VE  technological  feasibility  are  heavily  weighted  toward  leading  edge  VE 
computer  software  skills  and  development  facilities.  The  same  is  generally  true  for  teams 
pursuing  research  investigations.  In  fact,  for  these  teams  as  well  as  those  adapting  existing 
products,  there  is  likely  to  be  considerable  flexibility  with  regard  to  what  training  challenges  they 
take  -  what  training  domains  and  requirements  that  they  sign  up  for.  If  you  are  just 
demonstrating  a  technology,  then  it  will  probably  not  matter  much  if  the  chosen  application  is  for 
a  medical  procedure  or  for  operating  a  military  weapons  system,  so  long  as  the  demonstration 
effectively  illustrates  how  the  technological  features  might  serve  to  address  some  useful  training 
purposes.  However,  we  assume  that  in  the  normal  training  product  development  process,  the 
choice  of  application  area  is  not  so  arbitrary  -  the  training  requirement  is  typically  established 
completely  independently  from  the  potentially  relevant  technologies. 

For  VESTED,  we  envision  similar  user  communities  and  characteristics.  However,  we 
also  envision  a  very  broad  range  of  types  of  potential  users  for  both  products,  ranging  from  the 
low-echelon,  forward-deployed  individual  who  might  be  creating  some  “just  in  time”  training 
product,  such  as  to  train  unit  personnel  in  how  to  defeat  some  new  IED  variant  that  has  just 
appeared  in  the  local  area,  to  established  organizations  at  higher  echelon  levels  with  charters  for 
the  production  and  maintenance  of  widely  used  training  programs.  Thus,  the  user  may  be  a 
single  individual  performing  the  entire  production  effort  solitarily,  or  it  could  be  an  individual 
who  leads  or  works  within  a  larger  organization,  or  it  could  be  a  team  of  people  working  together 
with  the  VESTED  tool.  For  the  design  of  the  VESTED  tool,  it  is  particularly  important  to 
establish  the  kinds  of  relevant  expertise  that  the  user  or  user  team  will  or  won’t  possess.  At  least 
five  general  types  of  expertise  must  be  considered: 
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•  Training  pedagogy 

•  Subject  matter  expertise  in  the  target  skill  area 

•  Training  product  production  skills 

•  Artistic  skills  (manual  art,  digital  art,  photography,  etc.) 

•  Software  skills  (especially  in  VE,  multimedia  tools,  etc.) 

We  expect  that  most  users  will  have  some  expertise  in  the  area  of  training  pedagogy,  with 
a  good  understanding  of  the  training  objectives  for  the  intended  product  and  how  it  might  fit  into 
any  larger  training  program.  However,  we  expect  that  there  may  be  wide  variability  with  regard 
to  whether  or  not  the  users  have  some  or  little  expertise  in  the  other  areas  where  they  may  need 
to  draw  on  other  sources  within  or  outside  of  their  production  team. 


Design 


Because  of  its  focus  on  linear  demonstration  videos  rather  than  multiple  branching 
interactive  scenarios  and  its  consideration  of  modalities  other  than  VE  for  production,  VESTED 
must  provide  a  variety  of  functions.  Since  VESTED  addresses  only  the  design,  planning,  and 
storyboarding  phase  of  application  production  and  not  the  actual  video  creation,  it  does  not  get 
entangled  with  the  problem  of  a  complex  boundary  between  its  functionality  and  that  of  the  other 
video  production  tools  that  might  be  required.  Still,  there  is  a  considerable  challenge  to  the  user 
to  determine  how  to  pick  and  choose  among  the  various  relevant  media  (i.e.,  VE,  photography, 
manual  artwork,  music,  narration)  in  order  to  produce  an  adequate  training  product  that  will 
satisfy  training  objectives  and  production  constraints.  This  challenge  is  compounded  by  the 
complex  options  that  are  often  posed  by  availability  of  various  kinds  of  legacy  materials  in  some 
of  the  media  which  may  address  some  of  the  production  needs  to  varying  degrees.  A  further 
related  issue  is  the  consideration  of  the  potential  value  to  other  future  training  projects  and 
products  of  the  results  of  the  current  effort,  which  is  especially  important  for  cases  where  an 
elaborate  VE  product  is  to  be  constructed. 

Although  the  VESTED  user  will  not  be  required  to  work  in  a  strictly  linear,  sequential 
fashion  in  developing  the  specification  and  evaluations  of  options  for  training  demonstrations, 
we  will  assume  a  default  linear  sequence  for  our  description  of  the  VESTED  interface  design. 

We  expect  that  most  users  would  follow  roughly  this  sequence  for  their  initial  specifications  of 
their  demonstration  requirements  and  ideas,  but  we  also  expect  considerable  idiosyncratic 
iteration  following  the  “first  pass”  of  specifications  and  evaluations  in  order  to  adjust  and  refine 
the  initial  demonstration  plans  and  resolve  problems  identified  in  the  evaluation  results. 

The  primary  interface  screens  and  their  presumed  sequence  of  initial  use  are  as  follows: 

•  Specifying  requirements  and  linking  to  general  guidelines 

•  Objectives  specification 

•  Demonstration  outline 

•  Connecting  Objectives  to  Outline 

•  Case  construction  &  benefit  estimation 
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•  View  benefit  evaluations 

o  Tabular  display  of  single  case  benefits 
o  Graphic  display  of  single  case  benefits 
o  Tabular  display  of  differences  in  benefits  between  two  cases 
o  Graphic  display  of  differences  in  benefits  between  two  cases 

•  Create  and  view  cost  estimates 

o  Specify  foundation  elements  for  each  case 
o  Specify  cost  element  details  for  each  case 
o  View  cost  summary  for  each  case 
o  Specify  future  reuse  profile  for  each  case 
o  View  cost  projection  for  each  reuse  profile 

The  six  major  headings  in  the  above  outline  of  VESTED  interface  screens  represent  the 
six  distinct  functional  areas  through  which  VESTED  aids  the  user  in  specifying,  evaluating,  and 
deciding  on  a  plan  for  construction  of  a  training  demonstration.  The  user  begins  with  the 
specification  of  training  requirements  and  objectives  and  with  formulation  of  an  outline  for  the 
training  demonstration,  with  these  two  steps  being  optionally  perfonned  in  either  order.  Both  of 
these  first  two  steps  must  be  completed  before  the  third  and  forth  steps  of  “connecting  objectives 
to  outline”  and  “case  construction  &  benefit  estimation”  can  be  accomplished,  and  the  third  step 
must  precede  the  fourth.  Although  this  fourth  step  is  logically  really  a  composite  of  two  distinct 
functions,  it  is  represented  as  a  single  step  in  the  interface  because  it  seems  most  efficient  for  the 
user  to  specify  the  estimated  benefits  for  each  of  the  most  detailed  segments  of  the  storyboard 
outline  just  as  the  concept  for  media  implementation  of  that  segment  is  conceived  in 
specification  of  the  storyboard  outline.  The  final  two  functional  areas,  which  again  can  be 
addressed  in  either  order,  are  viewing  of  evaluation  results  and  creation  and  viewing  of  cost 
estimates. 

In  the  following  subsections,  we  describe  the  interface  screens  associated  with  each  of 
these  functional  areas. 

Specifying  requirements  and  linking  to  general  guidelines.  Although  we  have  determined 
that  the  current  research  literature  does  not  seem  to  support  the  compilation  of  any  very  detailed 
design  guidelines  for  the  design  of  training  demonstrations,  we  have  identified  a  few  general 
guidelines  (as  discussed  above  in  “Process  for  Guideline  Development”)  that  are  driven  by  some 
high  level  and  easily  identified  characteristics  of  the  training  requirement  and  target  tasks. 

Figure  10  illustrates  one  of  these  screens. 
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Figure  10.  Specification  of  general  requirements  and  links  to  high-level  guidelines. 


In  the  current  stage  of  VESTED  tool  implementation,  it  is  intended  that  these  guidelines 
are  to  be  integrated  into  the  design  process  through  the  discretion  and  ingenuity  of  the  user,  with 
no  direct  linkage  within  the  VESTED  software  between  these  initial  specifications  and  later 
stages  of  detailed  design  screens.  However,  it  is  our  expectation  that  experience  and  user 
feedback  from  initial  VESTED  applications  will  enable  us  to  formulate  increasingly  detailed 
guidelines  along  with  direct  integration  of  those  guidelines  into  many  of  the  screens  addressing 
detailed  design  issues. 

While  we  have  determined  that  formulation  of  predefined  detailed  guidelines  is  currently 
an  elusive  goal,  we  believe  that  it  is  important  for  the  user  to  develop  detailed  connections 
between  the  training  objectives  for  the  demonstration  and  the  design  features  of  the 
demonstration.  This  is  accomplished  by  requiring  that  the  user  specify  those  detailed  training 
objectives  before  developing  the  design.  The  VESTED  user  starts  by  defining  training 
objectives  in  accordance  with  a  standard  hierarchy  of  terminal  training  objectives  (highest,  most 
general,  and  most  expansive  level),  subordinate  training  objectives  (middle  level),  and  enabling 
objectives  (lowest  level  and  most  specific).  The  user  is  also  asked  to  classify  each  enabling 
objective  in  terms  of  a  small  number  of  predefined  categories  (see  Figure  11).  In  particular,  the 
following  major  categories  of  training  objective  have  been  identified  as  relevant: 
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•  Types  of  knowledge: 

o  Declarative  knowledge 
o  Procedural  knowledge 
o  Conceptual  knowledge 
o  Recognitional  or  perceptual  knowledge 

•  Moderators  of  knowledge  processing 

o  Integration  of  separate  knowledge  elements  into  an  interconnected  whole 
o  Link  to  prior  knowledge/experience 

•  Moderators  of  training  dynamics 

o  Orientation  or  focusing  of  the  trainee’s  attention 
o  Motivation 


Figure  11.  Specification  of  training  objectives. 


Demonstration  outline.  VESTED  requires  the  user  to  specify  an  outline  for  the  needed 
demonstration  in  the  fonn  of  a  storyboard  structure  which  will  serve  subsequently  as  the  basis  of 
decisions  about  what  media  and  what  legacy  materials  to  employ  for  application  production. 
Storyboards  are  conceived  as  a  sequence  of  elements  constructed  within  the  hierarchical 
organization  of  sections  (highest  level),  frames  (middle  level),  and  segments  (most  detailed 
level),  with  each  at  the  segment  level  tied  directly  to  at  least  one  specific  training  objective  that  is 


42 


designed  to  address  the  higher  level  training  goals  for  the  product.  Figure  12  represents  the 
Demonstration  Outline  screen.  It  offers  the  user  a  collection  of  standard  options  for  the  outline 
consisting  of  the  following: 

•  Titles 

•  Credits 

•  Objectives  overview 

•  Tutorial  overview 

•  Full  procedure  run-through 

•  Drill-down  of  procedure  steps 

•  Drill-down  of  procedure  variations 

•  Drill-down  of  special  conditions 

•  Drill-down  of  common  errors 

•  Review  of  full  procedure 

•  Take-away  points 

•  Summary 

Although  we  would  generally  expect  titles  to  go  first  and  summary  to  come  last,  it  is  not 
hard  to  imagine  all  sorts  of  orderings  of  these  pieces  for  different  types  of  material,  alternative 
training  objectives,  varying  target  audience  characteristic,  and  individual  training  styles. 
Accordingly,  the  user  is  allowed  to  select  from  and  order  these  elements  in  any  sequence  along 
with  creating  any  other  outline  heading  that  the  user  might  conceive  in  other  relevant  categories. 
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Figure  12.  Creating  the  demonstration  outline. 


Connecting  objectives  to  outline.  The  fundamental  principle  of  VESTED,  that  concepts 
for  training  demonstrations  be  driven  by  the  identification  of  specific  training  objectives, 
requires  that  at  least  one  objective  be  associated  with  each  sequential  segment  of  the  storyboard 
design  (which  consists  of  a  linear  sequence  of  segments  at  the  most  detailed  level  of 
representation  in  VESTED).  In  the  third  major  screen  and  functional  area  (see  Figure  13),  the 
user  is  asked  to  associate  one  or  more  enabling  objectives  to  each  segment  in  the  demonstration 
outline. 
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Figure  13.  Connecting  objectives  to  outline. 


Although  we  might  typically  expect  the  opening  section  and  frames  of  the  demonstration 
video  to  address  objectives  in  the  areas  of  orientation,  motivation,  and  linkage  to  prior  relevant 
knowledge,  the  user  must  be  allowed  to  associate  any  defined  objective  to  each  segment. 
Similarly,  we  might  expect  much  of  the  remainder  of  the  video  to  focus  on  the  various  kinds  and 
aspects  of  relevant  target  task  knowledge  -  declarative,  procedural,  conceptual,  recognitional, 
and  integrative,  but  some  segments  may  also  serve  primarily  to  address  moderating  objectives 
such  as  motivation  or  integration  with  prior  knowledge  (or  any  other  category  of  objective).  We 
could  identify  no  good  reasons  to  constrain  the  author  in  associating  any  types  of  objective  to 
any  portions  of  the  storyboard  sequence,  so  complete  flexibility  is  afforded  to  the  user  so  long  as 
at  least  one  objective  is  specified  for  each  segment.  But  multiple  category  types  may  be 
associated  with  each  storyboard  segment,  recognizing  that  multiple  distinct  objectives  may  be 
addressed  by  a  single  piece  of  simple  or  multi-modal  content  (e.g.,  with  a  visual  character 
displaying  the  performance  of  a  procedure  while  simultaneously  presenting  verbal  explanation  of 
declarative  information,  all  with  a  background  of  motivating  visual  and  auditory  imagery). 
However,  no  more  than  a  single  instantiation  of  each  distinct  type  of  objective  category  is 
permitted  within  a  single  segment.  For  example,  a  segment  could  have  two  separate  objectives 
representing  different  types  of  knowledge  (say  one  for  declarative  and  one  for  procedural)  but 
could  not  have  two  separate  objectives  in  the  same  category  (say  two  objectives  in  procedural 
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knowledge).  This  restriction  is  imposed  in  order  to  avoid  the  problem  of  creating  a  fonnula  for 
aggregating  individual  benefit  estimates  within  a  single  data  cell.  In  any  situation  where  the  user 
might  want  to  establish  multiple  separate  objectives  in  the  same  category  area  and  associate  them 
with  the  same  segment,  it  is  recommended  that  the  user  simply  define  a  new  composite  objective 
for  which  the  user  can  then  define  the  weight  and  estimated  benefits  separately  from  the  more 
elemental  objectives. 

For  each  individual  association  of  an  objective  with  a  storyboard  segment,  the  user  is 
further  required  to  select  an  importance  ‘weight’  for  that  instantiation  of  that  objective  according 
to  a  simple  three-point  scale  ranging  from  minimal  value  (=  1)  to  highest  possible  value  (=  3). 
These  weights  are  assigned  separately  for  each  segment  because  of  the  assumption  that  different 
segments  may  emphasize  the  same  objectives  differentially,  such  as  where  the  same  objective 
may  be  the  central  focus  of  one  segment  but  just  a  secondary  concern  in  another. 

At  the  conclusion  of  this  step  in  the  VESTED  process,  the  user  has  characterized  the 
detailed  requirements  for  the  training  demonstration  without  regard  to  how  that  requirement 
might  be  satisfied  through  an  implementation  of  a  training  product  in  some  combination  of 
media.  In  the  next  step,  the  user  will  construct  one  or  more  design  specifications  for  candidate 
alternative  plans,  or  ‘cases’,  for  the  needed  demonstration. 

Case  construction  &  benefit  estimation.  The  specifications  of  objectives  and  the 
demonstration  outline  then  serve  as  the  basis  for  the  further  development  of  descriptions  of  the 
requirements  for  video,  audio,  and  text  at  each  finest  (segment)  level  of  the  storyboard,  and  these 
requirements  in  turn  are  used  to  characterize  the  various  media  options  for  each  of  these 
elements,  including  the  possible  use  of  legacy  materials  as  well  as  the  construction  of  new 
material  in  any  of  the  pertinent  media  (see  Figure  14).  Each  complete  collection  of 
specifications  for  a  prospective  training  demonstration  is  tenned  as  a  ‘case’,  with  the  expectation 
that  typically  two  or  more  cases  will  be  constructed  to  evaluate  against  one  another  in  order  to 
determine  which  is  likely  to  be  most  successful  in  actual  implementation.  Our  goal  in  this 
process  is  to  enable  the  author  to  develop  the  generic  storyboard  as  quickly  and  efficiently  as 
possible  in  order  to  reach  an  infonned  decision  regarding  the  use  of  the  available  media  tools  and 
legacy  materials  which  can  then  be  instantiated  into  a  detailed  storyboard  and  production  plan. 
Before  proceeding  with  further  evaluation  of  costs  and  benefits  of  the  options,  the  author  is 
encouraged  to  identify  several  distinct  storyboard  ‘cases’  (such  as  one  in  which  all  imagery  is  in 
VE  mode,  another  that  is  all  photographic  video,  another  that  employs  exclusively  legacy 
imagery,  etc.). 
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Figure  14.  Case  construction  and  benefit  estimation  in  VESTED. 


We  found  that  there  has  been  considerable  experimental  research  regarding  the  training 
effectiveness  of  many  of  the  aspects  of  training  demonstration  design  (e.g.,  demonstration  of 
typical  errors,  repetition,  perspective,  etc.;  e.g.,  see  Wouters,  Paas  &  van  Merrienboer,  2008  and 
2009;  Riolo,  1997;  White  &  Hardy,  1995),  but  very  little  in  the  way  of  definitive,  generalizable 
results.  This  poses  a  special  challenge  for  VESTED  because  the  goal  of  VESTED  is  to  aid  the 
user  in  making  a  cost-benefit  decision  about  the  best  approach  for  construction  of  a  training 
demonstration.  While  the  estimation  of  production  costs  for  the  various  possible  media  and 
legacy  material  combinations  may  seem  like  a  manageable  challenge,  the  estimation  of  training 
benefits  for  each  case  is  considered  a  thornier  problem.  After  careful  investigation  of  both  of 
these  aspects  of  the  cost-benefit  trade-off,  we  have  concluded  that  an  analytically  guided  process 
of  subjective  estimation  is  most  suitable  in  both  areas. 

On  the  side  of  training  effectiveness  benefits,  our  approach  is  to  ask  the  user  to  estimate 
the  benefit  on  a  Likert-type  scale  for  each  segment  (smallest  unit)  of  the  storyboard  for  each 
objective  that  is  associated  with  that  segment.  Distinct  weights  were  previously  specified  for 
each  category  of  training  objective  (as  described  above  for  the  “connecting  objectives  to  outline” 
screen  and  functionality  area).  The  total  estimated  benefit  for  each  storyboard  segment  and 
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objective  category  is  calculated  as  the  product  of  the  specified  weight  and  the  estimated  benefit 
entered  by  the  user. 

View  benefit  evaluation.  VESTED  enables  the  user  to  view  weighted  benefit  estimates 
across  the  storyboard  timeline  with  the  primary  display  presenting  the  weighted  benefit  for  each 
category  of  objective  (separately)  across  the  timeline  of  the  specified  sequence  of  storyboard 
segments.  It  is  noteworthy  that  this  presentation  no  longer  differentiates  between  the  arbitrarily 
many  different  enabling  objectives  that  were  specified  in  the  demonstration  requirement,  but 
rather  the  results  for  all  objectives  in  an  objective  category  are  displayed  as  a  common  variable. 
These  estimated  measures  can  also  be  aggregated  across  sections,  frames,  and  segments,  with 
user-defined  weightings  applied  to  these  levels  and  the  results  presented  by  objective  category. 

It  is  important  to  be  able  to  view  the  benefit  estimation  data  for  each  individual  case  separately  in 
order  to  manage  and  iterate  the  estimation  process,  but  pairwise  comparison  of  pairs  of  cases  in 
terms  of  the  differences  between  benefits  for  the  two  cases  is  expected  to  be  most  useful  for 
making  selection  decisions  between  cases.  The  most  basic  single  case  display  of  results  is  the 
tabular  fonn  of  display  (see  Figure  15)  in  which  the  calculated  value  of  “weighted  benefit”  is 
presented,  possibly  with  color  coding  of  values  in  various  category  ranges  of  interest.  Those 
same  data  for  the  single  case  can  then  alternatively  be  viewed  in  graphic  form  (see  Figure  16). 
These  data  can  be  useful  in  aiding  the  training  product  developer  in  insuring  that  all  training 
objectives  are  adequately  addressed  by  any  give  design  case.  However,  in  order  to  make  the 
decision  as  to  which  of  several  design  cases  to  adopt,  it  is  also  helpful  for  the  developer  to  be 
able  to  view  the  complete  array  of  differences  in  benefit  estimates  between  any  two  design  cases. 
Such  decisions  between  multiple  cases  are  likely  to  be  frustrated  by  situations  where  some 
segments  and  objectives  produce  greater  benefits  for  one  case  and  other  cases  are  superior  for 
other  segments  and  objectives.  Thus,  by  viewing  the  pattern  of  differences  across  all  of  the 
estimated  benefits  between  two  selected  cases,  the  developer  can  be  expected  to  judge  which 
would  be  more  preferable  in  an  overall  sense.  To  aid  this  comparison,  VESTED  allows  the  same 
forms  of  data  presentation  used  for  single  cases  (Figures  15  and  16)  to  be  applied  for  the 
differences  in  benefit  values  for  any  two  cases  selected  by  the  user  (see  Figure  17  for  the  tabular 
view  and  Figure  18  for  the  graphic  view).  In  the  different  displays,  the  developer  need  only 
observe  the  sign  of  each  data  point  to  see  which  of  the  cases  is  superior  for  the  segment  and 
objective  that  it  represents. 
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Figure  15.  Tabular  display  of  weighted  benefit  evaluation  across  storyboard  timeline  for  one 
case. 
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Figure  16.  Graphic  display  of  weighted  benefit  evaluation  across  storyboard  timeline  for  one 
case. 
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Figure  17.  Tabular  display  of  weighted  benefit  evaluation  across  storyboard  timeline  for 
differences  between  two  cases. 
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Figure  18.  Graphic  display  of  differences  in  estimated  benefits  for  two  cases  in  VESTED. 


Create  and  view  cost  estimates.  VESTED  approaches  the  estimation  of  production  costs 
in  a  somewhat  different  fashion,  recognizing  that  all  production  costs  cannot  typically  be 
apportioned  incrementally  for  each  storyboard  segment,  but  rather  that  there  are  usually  some 
significant  monolithic  costs  that  apply  to  many  or  all  elements  of  the  storyboard,  which 
VESTED  designates  as  the  ‘foundation’  for  that  storyboard  case  (see  Figure  19). 
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Figure  19.  Specification  of  foundation  elements. 

For  example,  there  are  the  costs  of  buying  or  renting  production  equipment,  including 
software  packages  for  various  aspects  of  VE  or  photographic/video  processing  and  purchase  of 
relevant  legacy  components  that  may  be  commercially  available.  However,  after  accounting  for 
all  of  these  costs  that  cut  across  segments,  the  user  must  estimate  the  remaining  costs  that  will  be 
required  to  produce  the  imagery  for  each  individual  segment  (see  Figure  20). 
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Figure  20.  Specification  of  cost  estimation  details. 

While  some  of  these  costs  will  be  well-defined  monetary  costs  for  established 
commercial  products,  others  will  probably  be  in  terms  of  estimated  labor  hours  for  in-house  or 
outsourced  production  work,  though  estimated  hourly  labor  rates  can  easily  be  used  to  convert 
the  results  into  aggregated  monetary  units.  A  summary  of  all  costs  for  each  case  can  then  be 
combined  and  viewed  (see  Figure  21),  initially  for  each  isolated  case  and  then  for  comparisons  of 
costs  across  cases. 
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Figure  2 1 .  Summary  of  estimated  costs. 

While  the  costs  can  typically  be  condensed  down  to  a  single  dollar  figure  encompassing 
all  cost  elements,  benefits  are  necessarily  presented  more  complexly  as  profiles  of  the  various 
objective  categories  across  the  storyboard  timeline.  The  determination  of  how  to  weigh  costs 
against  benefits  in  each  situation  is  necessarily  left  to  the  user,  though  it  is  expected  that  there 
will  be  occasions  where  either  costs  or  benefits  are  very  nearly  equal  between  cases  of  interest  so 
that  the  decision  can  be  driven  by  the  one  criterion  that  differs  across  cases. 

There  is  yet  another  dimension  of  the  cost-benefit  consideration  that  can  serve  to 
distinguish  VE  production  efforts  from  other  alternatives  -  the  possible  future  uses  of  the 
planned  products.  Of  course,  any  training  demonstration  video  can  potentially  be  used  in  various 
other  training  contexts  than  the  initially  intended  site,  and  photographic  video  and  manual 
artwork  can  be  repurposed  piecemeal  in  future  productions  for  all  sorts  of  related  applications. 
However,  there  is  a  special  transitional  value  for  VE  facilities  because  of  their  typically 
expansive  adaptability.  For  example,  if  we  wanted  to  change  some  of  the  details  in  how  to  train 
Army  medics  in  perfonning  the  cricoidthyrotomy  procedure  at  some  future  time,  we  would 
probably  have  to  shoot  new  photographic  video  in  order  to  adapt  a  photo/video  product,  whereas 
it  would  probably  be  possible  with  relatively  minor  effort  and  cost  to  modify  the  behaviors  of  the 
pertinent  avatars  to  exhibit  the  revised  procedure.  While  it  may  be  difficult  to  anticipate  future 
requirements  where  such  adaptation  may  be  beneficial,  and  it  will  always  be  important  to 
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appropriately  discount  the  uncertainties  associated  with  those  speculations,  such  opportunities 
for  future  reuse  are  expected  to  constitute  significant  decision  criteria  in  some  cases. 

Thus,  the  separation  of  foundation  costs  from  storyboard-segment  costs  is  based  on  our 
expectation  of  the  potential  reuse  of  portions  of  an  initial  demonstration  product  to  address 
subsequent  training  requirements.  A  key  expectation  in  this  area  is  the  likelihood  that  it  would 
be  much  less  costly  to  adapt  an  initial  VE  product  for  subsequent  similar  training  applications 
than  would  be  the  case  for  a  conventionally  produced  video  demonstration.  Although  there  is  no 
current  compelling  empirical  evidence  to  support  this  expectation,  there  is  also  no  good  reason  to 
reject  it  either.  This  expectation  of  a  VE  reuse  advantage  is  based  on  the  idea  that  there  is  a 
collection  of  foundational  or  intermediary  products  for  VE  projects  (i.e.,  virtual  models  for 
objects,  characters,  dynamics,  etc.)  that  can  be  reused  much  more  extensively  and  readily  in  the 
VE  case  than  for  any  analogous  foundational  elements  for  other  media,  especially  relative  to  the 
medium  of  conventional  video  production  with  live  actors.  Accordingly,  VESTED  defines  the 
foundation  for  a  production  case  as  all  of  the  efforts,  materials,  and  related  costs  that  are 
independent  of,  and  usually  precedent  to,  the  creation  of  the  individual  segment  products.  Thus, 
for  conventional  video  production,  the  foundation  would  include  access  to  all  of  the  necessary 
cameras,  microphones,  recording  and  processing  equipment,  sets,  etc.,  while  the  corresponding 
foundation  for  VE  would  involve  construction  of  the  basic  VE  models  for  human  characters, 
tools  and  other  objects  that  are  central  to  the  intended  training,  and  background  environmental 
elements.  The  VESTED  user  is  asked  first  to  define  the  complete  collection  of  foundation 
requirements  (see  Figure  19)  for  each  production  case  of  interest  (recognizing  that  most  cases 
will  not  be  ‘pure’  representations  in  just  one  production  medium  but  will  rather  be  implemented 
as  hybrids  of  multiple  media). 

Following  the  specification  of  the  foundation  requirements,  the  user  estimates  the  cost  of 
creating  all  of  the  foundation  elements  in  all  relevant  media  production  categories,  and  then  the 
similar  estimates  for  the  costs  of  producing  the  required  segment  products  for  each  storyboard 
segment,  assuming  that  all  foundation  elements  are  available  to  support  efforts  for  each  segment. 
This  complete  collection  of  cost  estimation  requirements  is  illustrated  in  the  spreadsheet  format 
of  Figure  20  which  is  envisioned  for  eventual  implementation  in  VESTED  as  a  simpler,  more 
streamlined  sequence  of  multiple  interface  screens. 

In  order  to  estimate  the  costs  of  projected  future  reuses  of  an  initial  product  case,  the  user 
is  asked  to  make  separate  estimates  of  the  percentage  of  modifications  that  will  be  required  to  the 
foundation  portions  and  the  segment-specific  portions  of  each  case  being  considered,  along  with 
a  separate  estimate  of  the  likely  lag  time  until  the  production  of  the  new  product.  These 
estimates  may  be  made  for  several  distinct  subsequent  production  efforts  being  considered  (e.g., 
later  adaptations  of  an  initial  demonstration  to  some  closely  related  training  domains,  or  periodic 
updates  to  the  initial  demonstration  to  reflect  changing  technology).  In  order  to  calculate 
expected  costs  and  potential  benefits  of  one  case  over  another,  the  VESTED  tool  can  calculate 
basic  costs  for  the  new  products  as  the  estimated  costs  of  the  original  components  multiplied  by 
the  estimated  modification  percentages  and  then  discounted  and/or  inflated  according  to  the  lag 
time  before  the  product  is  planned  (see  Figure  22). 
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Figure  22.  Estimation  of  projected  reuse  costs. 
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VESTED  Software  Design 


The  VESTED  system  design  includes  three  specific  components,  each  mapped  to  one  of 
the  demonstration  processes  described  in  the  VESTED  conop  including:  a  demonstration 
specification  tool  which  enables  a  demonstration  author  to  define  instructional  storyboards 
utilizing  an  objective-based  guideline-driven  process,  a  video  generation  tool  which  utilizes  a 
variety  of  game-based  engines  to  facilitate  the  instantiation  of  the  storyboards  in  the  form  of  raw 
videos,  and  a  video  mixing  tool  which  enables  a  demonstration  author  to  mix  the  raw  videos, 
using  the  VESTED  guidelines,  and  thereby  produce  instructionally-meaningful  demonstration 
videos.  The  VESTED  approach  relies  on  a  concept  whereby  a  range  of  video-generation  tools 
(i.e.,  “game-engines”)  can  be  utilized  within  the  overall  VESTED  solution  depending  on  the 
specific  demonstration  requirements  identified  by  the  author.  The  overall  vision  for  the 
VESTED  functional  architecture  is  depicted  in  Figure  23. 


VESTED  Software  Concept 

The  central  piece  of  the  VESTED  system  is  the  authoring  toolset  which  provides  the 
main  portal  through  which  the  demonstration  author  identifies  training  objectives,  creates 
storyboards,  defines  storyboard  alternatives  (i.e.,  cases),  and  performs  cost-analyses  of  multiple 
cases.  The  authoring  and  analysis  tool  is  designed  to  function  as  a  standard  desktop  application. 
The  primary  function  of  the  VESTED  system  is  to  provide  the  author  with  an  organizing  tool  to 
aid  them  in  making  the  decisions  about  where  and  how  to  use  the  VE  medium  and  all  of  the 
relevant  other  authoring  tools.  The  basic  concept  of  VESTED  is  to  offer  a  tool  to  aid  the  author 
in  articulating  the  objectives,  requirements,  and  constraints  for  a  planned  training  demonstration, 
then  to  guide  them  through  the  development  of  one  or  several  storyboard  ‘cases’  that  would 
satisfy  those  conditions,  and  finally  to  structure  the  cost-benefit  decision  of  selecting  a  preferred 
case  and  elaborating  its  specification  into  a  detailed  production  plan.  Specifically,  the  VESTED 
authoring  tool  provides  authoring  across  four  functional  areas  including: 

•  purpose/characteristics/guidelines, 

•  training  requirements  definition, 

•  training  benefits  analysis, 

•  training  product  cost  analysis. 

The  VESTED  demonstration  specification  tool  was  constructed  as  a  stand-alone  desktop 
application  and  was  written  in  C#  with  Microsoft’s  .NET  framework.  We  leveraged  both  the 
Windows  Presentation  Foundation  (WPF)  and  Language  Integrated  Query  (LINQ)  components 
within  the  .NET  framework. 
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Demo  Specification 


Video  Mixing 


Figure  23.  VESTED  functional  system  design  vision. 

The  full  VESTED  vision  would  utilize  a  commercial  game  engine  system  to  provide  a 
virtual-environment  for  creating  3D  representations  of  authored  demonstrations  (as  opposed  to 
the  standard  tabular  storyboard  entry).  Basic  placement  and  editing  of  the  scenario  entities 
would  be  accomplished  via  the  existing  editing  capabilities  of  the  game  system.  For  any  given 
engine,  there  may  be  additional  components  which  would  need  to  be  built  to  provide  additional 
capabilities  to  that  particular  game  engine  (e.g.,  entity  behaviors).  A  number  of  criteria  are 
important  in  the  examination  of  commercially  available  game  engines  to  support  VESTED 
requirements,  such  as  malleability,  usability  of  graphical  user  interfaces,  capabilities/limitations, 
and  deployability. 
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